File Uploads

Wednesday, September 1st, 2010|Misc, Version 3|by Romans

For a while it was difficult to use file uploads in Agile Toolkit. Finally a proper implementation of the control is ready.

Theory About File uploads

Proper implementation of file uploads is difficult on several levels:

  • files should be uploaded independent from form upload. One reason is that noone would want to re-upload file if other fields contain validation errors. Uploads are also not working with AJAX form submission
  • in multi-user system, users must not see each other files.
  • in some cases you will want to have ability to have multiple file uploads in your form.
  • each file being uploaded contains some meta information – original filename, mime type, size and where it is located on the server.
  • it should be possible to upload files in the forms which add entries. In other words – if you create entry in the database you can’t link it with file table yet.
  • cramming files into single directory quickly turns into something too terrible
  • storing files into multiple locations (volumes) is required in project with heavy file use, but is not easy.
  • uploaded files must be validated by type, sometimes content.
  • multiple file upload and progress indication is often in demand.
  • FINALLY: file upload is a pain in every project.

Agile Toolkit: Virtual File Storage (Filestore)

We have implemented a Filestore system. It consists of 3 models: File, Volume and Type. Each sits in it’s own table in MySQL. So whenever new file is being uploaded – new entry is also created in MySQL. The idea comes from our work on Podcast hosting service (4 years ago), but the implementation is new and much more elegant.

Volume stores information about physical location, size limit and amount of files in the volume. Each volume creates subdirectories from 0 to ff (256 in total) and stores 4000 files in each directory. When all 256 buckets fill up, it will cycle through all of them adding by one file into each directory.

Type is defined as mime/type. When file is being uploaded, it attempts to find the right Type. If type does not exist, upload is not allowed.

File models store all the information about original file such as filename, size, mime type, etc.

Those types are basic ones, and they can be inherited and enhanced. For instance you might want to add more columns into file table and define those columns inside File model. Columns such as user_id, which will determine file ownership.

Filestore is implemented as an addon and it is available for current version of toolkit. Please get in touch with us if you are interested in using it.

Upload field

Upload is being implemented as form field. This is improvement from previous implementation and is located in trunk/lib/Form/Field/upload.php. File contains detailed documentation inside which will explain:

  • how can you use one of three modes: simple upload, flash-upload and iFrame upload. iFrame upload is preferred method to use at this time. flash-upload uses Uploadify (see my earlier post about it)
  • how you can limit number of files you want users to allow uploading, such as single-file upload or no more than 5 files.
  • how can you assign Controller to the field. Without controllers you will need to handle files yourself. Controller_Filestore_File works with upload files out of the box.

To look into more details of field implementation – it uses jQuery UI widget to manage uploads. Initially it will show one upload field. When you select a file there, event is triggered (onchange) and upload starts right away. While upload is happening you will see another upload field, where you can select another file thus starting yet another upload. When upload is complete, file is registered in the database and through JavaScript it is added into the list of uploaded files.

Additionally ID(s) of Filestore_File are piling up in the hidden field. This field value will be used when you submit the form. For a single file you can use foreign key+int. For multiple files you will need varchar.

Show uploaded files

Upload field relies on the template view/uploaded_files.html to show list of files you have successfully uploaded. Table is properly updated when you upload files or load form which already have files. It also allows you to delete files. You can even customize template and output your own fields from your own model.

Example

Adding upload field is as easy as this:

[php]
$f->addField(‘upload’,'file’)
->setController(‘Controller_Filestore_File’);
[/php]

Note that in multi-user environment you need to define your own controller which uses setMasterField to filter only files available to your users.

Linking files through intermediate table

addRelatedEntity functionality of Model/Table.php allows you to link multiple tables together. It will allow you to introduce intermediate table which links filestore_file with your own table. Here is example, which links files with table ‘document’ through table ‘document_attachment’:

[php]
$c=$this->add(‘Controller_Filestore_File’);

$c->addRelatedEntity(‘da’,'document_attachment’,
‘filestore_file_id’,'inner’,'related’,true);
$c->newField(‘version’)
->relEntity(‘da’,'version’)
;
$c->newField(‘document_id’)
->defaultValue($_GET['id']) // this is document.id
->relEntity(‘da’,'document_id’)
;
$c->addCondition(‘document_id’,$_GET['id']);
$u=$f->addFieldPlain(‘upload’,'file’);
$u->setController($c);
[/php]

In this case you should make sure that document already exists when you upload files and you don’t even need to store contents of the field. You don’t even need submit button on such a form as files are automatically associated once uploaded.

Thanks

I’m sending thanks to MVS for coming up with original idea. Also thanks to Jancha for help in development and testing.

Code of upload component is located in trunk and will be available in next release. Please help me test and improve the component.

http://firefly.activestate.com/romaninsh/atk4/wiki/WikiStart

3 Comments

romans
Posted September 6, 20104:13 pm

Initial uploader wouldn’t work in firefox due to few quirks

1) firefox will not allow to initialize iframe’s name dynamically. $(‘<iframe name=”blah”/>’).appendTo() simply won’t work. Needed to workaround through non-standard innerHTML

2) firefox will copy value of a <input type=”field”> when you clone it through this.element.clone(). Other browsers don’t do that. Besides javascript have no access to that field. So had to do workaround with innerHTMLs again.

Resulting code looks really terrible, but works. I’m wondering if uploader going to work in IE.

Svetlozar Kondakov
Posted September 24, 20108:32 pm

Lets hope code will be fixed soon!

This is a very elegant solution! I want to see it in action :) Is it possible to prepare a demo on this?

romans
Posted September 25, 20101:14 am

Yeah, I’ll prepare a demo soon. I have a grand plan regarding documentation ;) Stay tuned.