Updating large lists eg for AVETMISS or Postcodes

Friday, Feb 8, 2019

Note that this is performed by CourseSales.com administration staff NOT by clients

Also created a way to load new records from the browser for lists that are maintained in the database:

  1. Log in using the super user
  2. Go to /referencelist/upload
  3. Choose the type from the list - only certain lists with large numbers of records are maintained in the database
  4. File must be CSV with two columns - KEY, VALUE - and no heading
  5. Batch size can be 100 - 200, but don’t choose a larger batch size
  6. Choose how many batches - ie a batch size of 100 and a number of batches of 2 means 200 records will be processed (this is all to manage large file sizes)
  7. Start offset - eg start from batch 3
  8. Process changes is whether to download a CSV of what new records would be imported, or to process the new records directly into the database. Always leave UNTICKED to RUN A TRIAL BEFORE A REAL IMPORT. Save the downloaded file (which will tell you which records will be ignored and which imported, as well as the number of records that will be imported).

The process will compare the list to what is already in the database and work out which records are new. It will not delete any records. It will only add records if Process Changes is ticked.

Internal lists:

  1. *1 ANZSCO (AVETMISS 7.0 ANZSCO)
  2. *2 ANZSIC (AVETMISS 7.0 ANZSIC)
  3. *3 AVETMISS 70 Field of Education 6 (AVETMISS 7.0 Subject Field of Education - 6 digits NOT AVETMISS 7.0 Program Field of Education - 4 digits) - ASCED field of education classification
  4. #1 AVETMISS 7.0 programs (AVETMISS 7.0 Program Recognition Id):
  5. #2 AVETMISS 7.0 units (AVETMISS 7.0 Subject Unit)
  6. Field Export Ids
  7. AVETMISS 7.0 Postcodes (AVETMISS 7.0 Postcode) https://portal.usi.gov.au/org/ReportFile
  8. #3 Country list.

*1 Download here: http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/1220.02006?OpenDocument *2 Download here: http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/1292.0.55.0022006?OpenDocument *3 Download here: http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/1272.02001?OpenDocument #1 Download here: https://training.gov.au/Reporting/ReportInfo?reportName=AccreditedCourse (with modification of spreadsheet) #2 Download here: https://training.gov.au/Reporting/ReportInfo?reportName=UnitClassification (with modification of spreadsheet) #3 Download here: https://www.ncver.edu.au/rto-hub/statistical-standard-software/country Other lists of interest (country, language, occupation): https://www.ncver.edu.au/rto-hub/avetmiss-systems-files

Notes about ASCED

Subject field of education identifier is based on the field of education (FOE) at the detailed level (6-digit), which is one part of the Australian Standard Classification of Education (ASCED), ABS catalogue no.1272.0, 2001. - Included in the ‘AVETMISS 70 Field of Education 6’ field set to update Program field of education identifier is a code that identifies the subject matter that is the ultimate aim of the skills and knowledge gained in a qualification, course or skill set. The Program field of education identifier is based on the field of education (FOE) at the narrow level (4-digit), which is one part of the Australian Standard Classification of Education (ASCED), ABS catalogue no.1272.0, 2001. - Not included, a separate internal list that is not available for update.

Postcode

Note that for the postcode the list used by NCVER contains historical data no longer listed by Australia Post and specific postcode records not listed by Australia Post as shown below:

  • @@@@ - Not specified (not entered into large list)
  • 0000 - Postcode unknown or not available (not entered into large list)
  • OSPC - Overseas address

Download the postcode list here: https://portal.usi.gov.au/org/ReportFile

Change the rows to match the “Key, Value” format:

 0200,"AUSTRALIAN NATIONAL UNIVERSITY",ACT,"PO BOXES","Post Office Boxes"

Modify to be:

 0200,AUSTRALIAN NATIONAL UNIVERSITY ACT
Note
In the postcode list the columns ‘Comment,Category’ should be removed. As should the comma seperating the locality from the state, which should be after the last space
Note
The list includes all postcodes, new and old; PO Box and delivery; Large Volume Receiver (LVR) and OSPC values (but not @@@@ nor 0000)

The following have been added:

 OSPC, Other Australian territories or dependencies ATD
 OSPC, Other (Overseas but not an Australian territory or dependency) OVR

(the ATD and OVR are indicators for the autocomplete script to allocate the correct state)

Points to note

  • The import only checks the KEY of the records not the VALUE (decodes) - so it looks for changes in the KEY, and ignores if the key exists, even if the VALUE has changed.
  • Only certain lists with large numbers of records are maintained in the database
  • File must be CSV with two columns - KEY, VALUE - and no heading
  • “Batch size” can be 100 - 200, but don’t choose a larger batch size
  • “Limit to this many batches” Choose how many batches - ie a batch size of 100 and a number of batches of 2 means 200 records will be processed (this is all to manage large file sizes)
  • “Start offset” - eg start from batch 3
  • “Process changes?” is whether to not upload but download a CSV of the new records identified by the process, or to process the new records into the database.
  • It will not delete any records. It will only add records if “Process changes?” is checked.
  • ALWAYS RUN A TRIAL BEFORE A REAL IMPORT and scan the proposed changes.
  • Ensure that the csv includes quotes too - as some of the VALUEs include commas.
  • Before the summary of the records added look for ‘File had 6521 rows’ and confirm this matches the number of records in the original import file.