mirrors/django
1
0
Fork 0
mirror of https://github.com/django/django.git synced 2025-10-24 06:06:09 +00:00
Code Issues 32 Releases Wiki Activity

File storage refactoring, adding far more flexibility to Django's file handling. The new files.txt document has details of the new features.

Browse Source 
This is a backwards-incompatible change; consult BackwardsIncompatibleChanges for details.

Fixes #3567, #3621, #4345, #5361, #5655, #7415.

Many thanks to Marty Alchin who did the vast majority of this work.


git-svn-id: http://code.djangoproject.com/svn/django/trunk@8244 bcc190cf-cafb-0310-a4f2-bffc1f526a37
This commit is contained in:
Jacob Kaplan-Moss
2008-08-08 20:59:02 +00:00
parent c49eac7d4f
commit 7899568e01
33 changed files with 1585 additions and 457 deletions
Download Patch File Download Diff File Expand all files Collapse all files

39
docs/custom_model_fields.txt
View File

@@ -596,3 +596,42 @@ smoothly:
instance, not a ``HandField``). So if your ``__unicode__()`` method
automatically converts to the string form of your Python object, you can
save yourself a lot of work.
Writing a ``FileField`` subclass
=================================
In addition to the above methods, fields that deal with files have a few other
special requirements which must be taken into account. The majority of the
mechanics provided by ``FileField``, such as controlling database storage and
retrieval, can remain unchanged, leaving subclasses to deal with the challenge
of supporting a particular type of file.
Django provides a ``File`` class, which is used as a proxy to the file's
contents and operations. This can be subclassed to customzie hwo the file is
accessed, and what methods are available. It lives at
``django.db.models.fields.files``, and its default behavior is explained in the
`file documentation`_.
Once a subclass of ``File`` is created, the new ``FileField`` subclass must be
told to use it. To do so, simply assign the new ``File`` subclass to the special
``attr_class`` attribute of the ``FileField`` subclass.
.. _file documentation: ../files/
A few suggestions
------------------
In addition to the above details, there are a few guidelines which can greatly
improve the efficiency and readability of the field's code.
1. The source for Django's own ``ImageField`` (in
``django/db/models/fields/files.py``) is a great example of how to
subclass ``FileField`` to support a particular type of file, as it
incorporates all of the techniques described above.
2. Cache file attributes wherever possible. Since files may be stored in
remote storage systems, retrieving them may cost extra time, or even
money, that isn't always necessary. Once a file is retrieved to obtain
some data about its content, cache as much of that data as possible to
reduce the number of times the file must be retrieved on subsequent
calls for that information.

43
docs/db-api.txt
View File

@@ -2298,53 +2298,34 @@ For a full example, see the `lookup API sample model`_.
get_FOO_filename()
------------------
For every ``FileField``, the object will have a ``get_FOO_filename()`` method,
where ``FOO`` is the name of the field. This returns the full filesystem path
to the file, according to your ``MEDIA_ROOT`` setting.
.. note::
It is only valid to call this method **after** saving the model when the
field has been set. Prior to saving, the value returned will not contain
the upload directory (the `upload_to` parameter) in the path.
Note that ``ImageField`` is technically a subclass of ``FileField``, so every
model with an ``ImageField`` will also get this method.
**Deprecated in Django development version**; use ``object.FOO.name`` instead.
See `managing files`_ for details.
get_FOO_url()
-------------
For every ``FileField``, the object will have a ``get_FOO_url()`` method,
where ``FOO`` is the name of the field. This returns the full URL to the file,
according to your ``MEDIA_URL`` setting. If the value is blank, this method
returns an empty string.
.. note::
As with ``get_FOO_filename()``, it is only valid to call this method
**after** saving the model, otherwise an incorrect result will be
returned.
**Deprecated in Django development version**; use ``object.FOO.url`` instead.
See `managing files`_ for details.
get_FOO_size()
--------------
For every ``FileField``, the object will have a ``get_FOO_size()`` method,
where ``FOO`` is the name of the field. This returns the size of the file, in
bytes. (Behind the scenes, it uses ``os.path.getsize``.)
**Deprecated in Django development version**; use ``object.FOO.size`` instead.
See `managing files`_ for details.
save_FOO_file(filename, raw_contents)
-------------------------------------
For every ``FileField``, the object will have a ``save_FOO_file()`` method,
where ``FOO`` is the name of the field. This saves the given file to the
filesystem, using the given filename. If a file with the given filename already
exists, Django adds an underscore to the end of the filename (but before the
extension) until the filename is available.
**Deprecated in Django development version**; use ``object.FOO.save()`` instead.
See `managing files`_ for details.
get_FOO_height() and get_FOO_width()
------------------------------------
For every ``ImageField``, the object will have ``get_FOO_height()`` and
``get_FOO_width()`` methods, where ``FOO`` is the name of the field. This
returns the height (or width) of the image, as an integer, in pixels.
**Deprecated in Django development version**; use ``object.FOO.width`` and
``object.FOO.height`` instead. See `managing files`_ for details.
.. _`managing files`: ../files/
Shortcuts
=========

388
docs/files.txt Normal file
View File

@@ -0,0 +1,388 @@
==============
Managing files
==============
**New in Django development version**
This document describes Django's file access APIs.
By default, Django stores files locally, using the ``MEDIA_ROOT`` and
``MEDIA_URL`` settings_. The examples below assume that you're using
these defaults.
However, Django provides ways to write custom `file storage systems`_ that
allow you to completely customize where and how Django stores files. The
second half of this document describes how these storage systems work.
.. _file storage systems: `File storage`_
.. _settings: ../settings/
Using files in models
=====================
When you use a `FileField`_ or `ImageField`_, Django provides a set of APIs you can use to deal with that file.
.. _filefield: ../model-api/#filefield
.. _imagefield: ../model-api/#imagefield
Consider the following model, using a ``FileField`` to store a photo::
class Car(models.Model):
name = models.CharField(max_length=255)
price = models.DecimalField(max_digits=5, decimal_places=2)
photo = models.ImageField(upload_to='cars')
Any ``Car`` instance will have a ``photo`` attribute that you can use to get at
the details of the attached photo::
>>> car = Car.object.get(name="57 Chevy")
>>> car.photo
<ImageFieldFile: chevy.jpg>
>>> car.photo.name
u'chevy.jpg'
>>> car.photo.path
u'/media/cars/chevy.jpg'
>>> car.photo.url
u'http://media.example.com/cars/chevy.jpg'
This object -- ``car.photo`` in the example -- is a ``File`` object, which means
it has all the methods and attributes described below.
The ``File`` object
===================
Internally, Django uses a ``django.core.files.File`` any time it needs to
represent a file. This object is a thin wrapper around Python's `built-in file
object`_ with some Django-specific additions.
.. _built-in file object: http://docs.python.org/lib/bltin-file-objects.html
Creating ``File`` instances
---------------------------
Most of the time you'll simply use a ``File`` that Django's given you (i.e. a
file attached to an model as above, or perhaps an `uploaded file`_).
.. _uploaded file: ../uploading_files/
If you need to construct a ``File`` yourself, the easiest way is to create one
using a Python built-in ``file`` object::
>>> from django.core.files import File
# Create a Python file object using open()
>>> f = open('/tmp/hello.world', 'w')
>>> myfile = File(f)
Now you can use any of the ``File`` attributes and methods defined below.
``File`` attributes and methods
-------------------------------
Django's ``File`` has the following attributes and methods:
``File.path``
~~~~~~~~~~~~~
The absolute path to the file's location on a local filesystem.
Custom `file storage systems`_ may not store files locally; files stored on
these systems will have a ``path`` of ``None``.
``File.url``
~~~~~~~~~~~~
The URL where the file can be retrieved. This is often useful in templates_; for
example, a bit of a template for displaying a ``Car`` (see above) might look
like::
<img src='{{ car.photo.url }}' alt='{{ car.name }}' />
.. _templates: ../templates/
``File.size``
~~~~~~~~~~~~~
The size of the file in bytes.
``File.open(mode=None)``
~~~~~~~~~~~~~~~~~~~~~~~~
Open or reopen the file (which by definition also does ``File.seek(0)``). The
``mode`` argument allows the same values as Python's standard ``open()``.
When reopening a file, ``mode`` will override whatever mode the file was
originally opened with; ``None`` means to reopen with the original mode.
``File.read(num_bytes=None)``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Read content from the file. The optional ``size`` is the number of bytes to
read; if not specified, the file will be read to the end.
``File.__iter__()``
~~~~~~~~~~~~~~~~~~~
Iterate over the file yielding one line at a time.
``File.chunks(chunk_size=None)``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Iterate over the file yielding "chunks" of a given size. ``chunk_size`` defaults
to 64 KB.
This is especially useful with very large files since it allows them to be
streamed off disk and avoids storing the whole file in memory.
``File.multiple_chunks(chunk_size=None)``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Returns ``True`` if the file is large enough to require multiple chunks to
access all of its content give some ``chunk_size``.
``File.write(content)``
~~~~~~~~~~~~~~~~~~~~~~~
Writes the specified content string to the file. Depending on the storage system
behind the scenes, this content might not be fully committed until ``close()``
is called on the file.
``File.close()``
~~~~~~~~~~~~~~~~
Close the file.
.. TODO: document the rest of the File methods.
Additional ``ImageField`` attributes
------------------------------------
``File.width`` and ``File.height``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
These attributes provide the dimensions of the image.
Additional methods on files attached to objects
-----------------------------------------------
Any ``File`` that's associated with an object (as with ``Car.photo``, above)
will also have a couple of extra methods:
``File.save(name, content, save=True)``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Saves a new file with the file name and contents provided. This will not replace
the existing file, but will create a new file and update the object to point to
it. If ``save`` is ``True``, the model's ``save()`` method will be called once
the file is saved. That is, these two lines::
>>> car.photo.save('myphoto.jpg', contents, save=False)
>>> car.save()
are the same as this one line::
>>> car.photo.save('myphoto.jpg', contents, save=True)
``File.delete(save=True)``
~~~~~~~~~~~~~~~~~~~~~~~~~~
Remove the file from the model instance and delete the underlying file. The
``save`` argument works as above.
File storage
============
Behind the scenes, Django delegates decisions about how and where to store files
to a file storage system. This is the object that actually understands things
like file systems, opening and reading files, etc.
Django's default file storage is given by the `DEFAULT_FILE_STORAGE setting`_;
if you don't explicitly provide a storage system, this is the one that will be
used.
.. _default_file_storage setting: ../settings/#default-file-storage
The built-in filesystem storage class
-------------------------------------
Django ships with a built-in ``FileSystemStorage`` class (defined in
``django.core.files.storage``) which implements basic local filesystem file
storage. Its initializer takes two arguments:
====================== ===================================================
Argument Description
====================== ===================================================
``location`` Optional. Absolute path to the directory that will
hold the files. If omitted, it will be set to the
value of your ``MEDIA_ROOT`` setting.
``base_url`` Optional. URL that serves the files stored at this
location. If omitted, it will default to the value
of your ``MEDIA_URL`` setting.
====================== ===================================================
For example, the following code will store uploaded files under
``/media/photos`` regardless of what your ``MEDIA_ROOT`` setting is::
from django.db import models
from django.core.files.storage import FileSystemStorage
fs = FileSystemStorage(base_url='/media/photos')
class Car(models.Model):
...
photo = models.ImageField(storage=fs)
`Custom storage systems`_ work the same way: you can pass them in as the
``storage`` argument to a ``FileField``.
.. _custom storage systems: `writing a custom storage system`_
Storage objects
---------------
Though most of the time you'll want to use a ``File`` object (which delegates to
the proper storage for that file), you can use file storage systems directly.
You can create an instance of some custom file storage class, or -- often more
useful -- you can use the global default storage system::
>>> from django.core.files.storage import default_storage
>>> path = default_storage.save('/path/to/file', 'new content')
>>> path
u'/path/to/file'
>>> default_storage.filesize(path)
11
>>> default_storage.open(path).read()
'new content'
>>> default_storage.delete(path)
>>> default_storage.exists(path)
False
Storage objects define the following methods:
``Storage.exists(name)``
~~~~~~~~~~~~~~~~~~~~~~~~
``True`` if a file exists given some ``name``.
``Storge.path(name)``
~~~~~~~~~~~~~~~~~~~~~
The local filesystem path where the file can be opened using Python's standard
``open()``. For storage systems that aren't accessible from the local
filesystem, this will raise ``NotImplementedError`` instead.
``Storage.size(name)``
~~~~~~~~~~~~~~~~~~~~~~
Returns the total size, in bytes, of the file referenced by ``name``.
``Storage.url(name)``
~~~~~~~~~~~~~~~~~~~~~
Returns the URL where the contents of the file referenced by ``name`` can be
accessed.
``Storage.open(name, mode='rb')``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Opens the file given by ``name``. Note that although the returned file is
guaranteed to be a ``File`` object, it might actually be some subclass. In the
case of remote file storage this means that reading/writing could be quite slow,
so be warned.
``Storage.save(name, content)``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Saves a new file using the storage system, preferably with the name specified.
If there already exists a file with this name ``name``, the storage system may
modify the filename as necessary to get a unique name. The actual name of the
stored file will be returned.
``Storage.delete(name)``
~~~~~~~~~~~~~~~~~~~~~~~~
Deletes the file referenced by ``name``. This method won't raise an exception if
the file doesn't exist.
Writing a custom storage system
===============================
If you need to provide custom file storage -- a common example is storing files
on some remote system -- you can do so by defining a custom storage class.
You'll need to follow these steps:
#. Your custom storage system must be a subclass of
``django.core.files.storage.Storage``::
from django.core.files.storage import Storage
class MyStorage(Storage):
...
#. Django must be able to instantiate your storage system without any arguments.
This means that any settings should be taken from ``django.conf.settings``::
from django.conf import settings
from django.core.files.storage import Storage
class MyStorage(Storage):
def __init__(self, option=None):
if not option:
option = settings.CUSTOM_STORAGE_OPTIONS
...
#. Your storage class must implement the ``_open()`` and ``_save()`` methods,
along with any other methods appropriate to your storage class. See below for
more on these methods.
In addition, if your class provides local file storage, it must override
the ``path()`` method.
Custom storage system methods
-----------------------------
Your custom storage system may override any of the storage methods explained
above in `storage objects`_. However, it's usually better to use the hooks
specifically designed for custom storage objects. These are:
``_open(name, mode='rb')``
~~~~~~~~~~~~~~~~~~~~~~~~~~
**Required**.
Called by ``Storage.open()``, this is the actual mechanism the storage class
uses to open the file. This must return a ``File`` object, though in most cases,
you'll want to return some subclass here that implements logic specific to the
backend storage system.
``_save(name, content)``
~~~~~~~~~~~~~~~~~~~~~~~~
Called by ``Storage.save()``. The ``name`` will already have gone through
``get_valid_name()`` and ``get_available_name()``, and the ``content`` will be a
``File`` object itself. No return value is expected.
``get_valid_name(name)``
------------------------
Returns a filename suitable for use with the underlying storage system. The
``name`` argument passed to this method is the original filename sent to the
server, after having any path information removed. Override this to customize
how non-standard characters are converted to safe filenames.
The code provided on ``Storage`` retains only alpha-numeric characters, periods
and underscores from the original filename, removing everything else.
``get_available_name(name)``
----------------------------
Returns a filename that is available in the storage mechanism, possibly taking
the provided filename into account. The ``name`` argument passed to this method
will have already cleaned to a filename valid for the storage system, according
to the ``get_valid_name()`` method described above.
The code provided on ``Storage`` simply appends underscores to the filename
until it finds one that's available in the destination directory.

58
docs/model-api.txt
View File

@@ -224,26 +224,64 @@ set to 75 by default, but you can specify it to override default behavior.
``FileField``
~~~~~~~~~~~~~
A file-upload field. Has one **required** argument:
A file-upload field. Has two special arguments, of which the first is
**required**:
====================== ===================================================
Argument Description
====================== ===================================================
``upload_to`` A local filesystem path that will be appended to
your ``MEDIA_ROOT`` setting to determine the
output of the ``get_<fieldname>_url()`` helper
function.
``upload_to`` Required. A filesystem-style path that will be
prepended to the filename before being committed to
the final storage destination.
**New in Django development version**
This may also be a callable, such as a function,
which will be called to obtain the upload path,
including the filename. See below for details.
``storage`` **New in Django development version**
Optional. A storage object, which handles the
storage and retrieval of your files. See `managing
files`_ for details on how to provide this object.
====================== ===================================================
This path may contain `strftime formatting`_, which will be replaced by the
date/time of the file upload (so that uploaded files don't fill up the given
directory).
.. _managing files: ../files/
The ``upload_to`` path may contain `strftime formatting`_, which will be
replaced by the date/time of the file upload (so that uploaded files don't fill
up the given directory).
**New in Django development version**
If a callable is provided for the ``upload_to`` argument, that callable must be
able to accept two arguments, and return a Unix-style path (with forward
slashes) to be passed along to the storage system. The two arguments that will
be passed are:
====================== ===================================================
Argument Description
====================== ===================================================
``instance`` An instance of the model where the ``FileField`` is
defined. More specifically, this is the particular
instance where the current file is being attached.
**Note**: In most cases, this object will not have
been saved to the database yet, so if it uses the
default ``AutoField``, *it might not yet have a
value for its primary key field*.
``filename`` The filename that was originally given to the file.
This may or may not be taken into account when
determining the final destination path.
====================== ===================================================
The admin represents this field as an ``<input type="file">`` (a file-upload
widget).
Using a ``FileField`` or an ``ImageField`` (see below) in a model takes a few
steps:
Using a ``FileField`` or an ``ImageField`` (see below) in a model without a
specified storage system takes a few steps:
1. In your settings file, you'll need to define ``MEDIA_ROOT`` as the
full path to a directory where you'd like Django to store uploaded

10
docs/settings.txt
View File

@@ -426,6 +426,16 @@ Default content type to use for all ``HttpResponse`` objects, if a MIME type
isn't manually specified. Used with ``DEFAULT_CHARSET`` to construct the
``Content-Type`` header.
DEFAULT_FILE_STORAGE
--------------------
Default: ``'django.core.filestorage.filesystem.FileSystemStorage'``
Default file storage class to be used for any file-related operations that don't
specify a particular storage system. See the `file documentation`_ for details.
.. _file documentation: ../files/
DEFAULT_FROM_EMAIL
------------------

25
docs/upload_handling.txt
View File

@@ -155,25 +155,8 @@ Three `settings`_ control Django's file upload behavior:
``UploadedFile`` objects
========================
All ``UploadedFile`` objects define the following methods/attributes:
``UploadedFile.read(self, num_bytes=None)``
Returns a byte string of length ``num_bytes``, or the complete file if
``num_bytes`` is ``None``.
``UploadedFile.chunks(self, chunk_size=None)``
A generator yielding small chunks from the file. If ``chunk_size`` isn't
given, chunks will be 64 KB.
``UploadedFile.multiple_chunks(self, chunk_size=None)``
Returns ``True`` if you can expect more than one chunk when calling
``UploadedFile.chunks(self, chunk_size)``.
``UploadedFile.size``
The size, in bytes, of the uploaded file.
``UploadedFile.name``
The name of the uploaded file as provided by the user.
In addition to those inherited from `File`_, all ``UploadedFile`` objects define
the following methods/attributes:
``UploadedFile.content_type``
The content-type header uploaded with the file (e.g. ``text/plain`` or
@@ -186,13 +169,11 @@ All ``UploadedFile`` objects define the following methods/attributes:
For ``text/*`` content-types, the character set (i.e. ``utf8``) supplied
by the browser. Again, "trust but verify" is the best policy here.
``UploadedFile.__iter__()``
Iterates over the lines in the file.
``UploadedFile.temporary_file_path()``
Only files uploaded onto disk will have this method; it returns the full
path to the temporary uploaded file.
.. _File: ../files/
Upload Handlers
===============