Upload and convert

By: ragnvald on: Sun 13 of Dec., 2009 14:00 UTC (1095 Reads)

We are now testing out an upload and convert service on this website. This means that one of our pages will let you upload a .dbf-file with LON/LAT columns and then download a file with a QDGC string. We have dealt with some initial problems, and the service now seems to be up and running, although we might still have the odd error. In case the service, crashes there is a restart every hour.

After publishing the lonlat2qdgc client we have concluded that not everyone finds running the compiled python software to be the easiest of tasks. The conclusion was to run the conversion utility as a service on our web server.

This is where you get things done:

Upload and convert a .dbf-file

As mentioned this is currently undergoing testing. Programming and running a server side service has been challenging. We have decided on a model allowing users to upload their data file using a web page. This web page is always running. On the server we are running a script polling the incoming folder for new files every 15 second. New files are converted one at a time and placed in a dedicated folder whereupon an email with the unique link is sent to the user.

More detailed the conversion utility relies on the user uploading a .dbf-file (the data file) with the following columns:

DBF-format

LON - FLOAT
LAT - FLOAT
QDGC - STRING

A sample file is available for download here:

sample_lonlat2qdgc.dbf

This is the sample file viewed using OpenOffice?.org Calc:

The content of the lon/lat columns are mandatory. The QDGC string will be over written if available.

The code is available on a separate page:

Python code lonlat2qdgc daemon

To secure your data the link to your uploaded data will not be available for other users. The link is supplied to you in an email sent to the email address provided at upload time.

The current conversion routine might crash with the wrong parameters. This means that the system will shut down for half an hour or so before being reset. We are working to resolve this issue. The conversion utility converts around 5000 positions per seconds. If you have a big file - just bring it on! Once the conversion has finished you will receive an email with information similar to this one:

System feedback to user

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-
- Client email      : youremail@yourcompany.com
- Client QDGC level : 6
- User Comments     : This is a useful utility!
-
- Job was submitted : 2009-12-13 14:48.
-
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Conversion of 19999 posts done i 3.16 seconds. This is about 6328 conversions per second.


The link to your converted file is: http://www.qdgc.org/lonlat2qdgc/userdata/dgzRZqtYwJpxZWyChjmx/out/jondoe.dbf

The link to a log file is: http://www.qdgc.org/lonlat2qdgc/userdata/dgzRZqtYwJpxZWyChjmx/out/jondoe.log

This work is currently not funded. You might find a banner with google adverts on the site. Any money generated will be donated to www.tzcrc.org

Buglist:

Date processed is referred to as date submitted.
The dbfpy library used has some compatibility issues making the script conversion limit 65535 posts.