New topic Closed topic
avatar image
1
Import text file without header or add a header to it
By Created , last editted

Hi,

I have to import a fixed width text file that does not have a header. Using the standard import functionality will now skip the first row, because it does expect a header.

I tried to add a header row by using the read_file() expression. Using liquid I print the file content on a GET endpoint below a manually added header row. Then using a url expression and a url that ends on a file extension I save this rendered endpoint as a file.

I thought it was working, but I noticed that using this method, the file is not fixed with anymore. Records in the file are not equally distributed anymore. I tried different endpoint types, like HTML, CCS etc. I also added the "raw" option in the liquid tag. I is better but still records are not equally distributed.

I am planning to do a couple of additional tests like

  • Save file content in a multi-line text field just to see if the issues is caused by the expression itself or when it is rendered in an endpoint.
  • Try to create a collection based on de filecontent and process de data in a loop, but I believe that this wil not work.

If one on the above does work, than I have the possible issue that the files are too big to process like this, because the files vary from 1k to more dan 500K records.

Has any anyone also faced an issue like this, and have managed to solve this?

Thanks.


Hi,

I have to import a fixed width text file that does not have a header. Using the standard import functionality will now skip the first row, because it does expect a header.

I tried to add a header row by using the read_file() expression. Using liquid I print the file content on a GET endpoint below a manually added header row. Then using a url expression and a url that ends on a file extension I save this rendered endpoint as a file.

I thought it was working, but I noticed that using this method, the file is not fixed with anymore. Records in the file are not equally distributed anymore. I tried different endpoint types, like HTML, CCS etc. I also added the "raw" option in the liquid tag. I is better but still records are not equally distributed.

I am planning to do a couple of additional tests like

  • Save file content in a multi-line text field just to see if the issues is caused by the expression itself or when it is rendered in an endpoint.
  • Try to create a collection based on de filecontent and process de data in a loop, but I believe that this wil not work.

If one on the above does work, than I have the possible issue that the files are too big to process like this, because the files vary from 1k to more dan 500K records.

Has any anyone also faced an issue like this, and have managed to solve this?

Thanks.

Answers
Sort by:

This topic is closed.