I think everything is here (famous last words). Updates will be posted as necessary.
UPDATE: 10/14/99 ask you to provide an example of your difference matte routine, and a discussion of the quality requirements for matte extraction.
UPDATE: 10/11/99 clarify coordinate systems
UPDATE: 10/10/99 the instructions for turning in the assignment are not provided, as are the questions you must answer.
UPDATE: 10/10/99 a clarification on the initial READ operation. (go to the update)
UPDATE: 10/10/99 arbitrary scaling is a BONUS feature (correction)
Late assignments will be accepted (according the to project late policy) until Friday, October 22nd.
In this assignment, you will implement a basic image compositing system. This system will allow the user to combine a number of images into a single image.
Your program must work in two “modes:” interactive and batch. The batch mode reads the description of the compositing operations in from a file in a special form described below, while the interactive mode allows the user to specify the images and operations to be performed. Your user interface does not need to be fancy, but it will need to allow the user to access the features of your program.
NOTE: simply allowing the user to type in the batch commands (and pop up windows to display the intermediate results) will be an acceptable basic user interface.
We will provide you with some test images for your program. As you should know from class, the hard part in obtaining images for compositing is getting alpha mats. You will implement some methods for determining the alpha mats.
In interactive mode, your program prompts the user for a tiff file containing an image, and how to place it into the final result. The first image read in serves as the “canvas” and all the resulting images are placed onto it. For each image, you must ask the user what compositing operation to perform, whether or not to compute the alpha channel by blue-screening, how big to make the read in image (scaling it), and where the image should be placed on the canvas. At each step, your program should show the images read in, and the intermediate compositing results. You program must also be able to write the resulting composite back out to a TIFF file.
Batch mode simply records the same processes done interactively into a file to make it easier to test.
With the projects, we are purposely designing them to be a bit hard.
We don’t expect everybody to get all of the assignment to work. It’s like on an exam, we don’t expect everyone to get 100%. Of course, we don’t know how hard it will be for you beforehand, which is why there will be a curve involved.
To help you gauge how well you’re doing, we will group parts of the assignment description into three parts:
BASIC – we expect that everyone should be able to get these parts. We expect that if you get all of these parts to work, your grade will be about average.
ADVANCED – we expect that most people will get some of these parts. In order to get a good grade, you will need to
BONUS – these are more difficult parts, designed to be interesting extensions to the assignment. If you get the other parts of the assignment, you might try to do these to further your understanding of the concepts, show off how smart you are, or you’re your programming skills a workout. These will not count towards your grade.
In the project description, we will identify specific tasks as being Basis, Advanced, or bonUs (Unrequired). An oversimplified way to view this is that if you get all the B’s, you’ll probably get a B, if you get all the A’s (and B’s) you’ll probably get an A. We will provide an exact scoring sheet later. (we have not decided whether to publish this before you turn in your assignment).
You should also remember, that your grade will be based on 3 parts:
This assignment asks you to construct a fairly sizeable system. There's a good chance this may be the biggest program you've ever written - especially if you're one of the people who have only built programs for intro classes. Even if you're a pro, implementing everything in the 3 week time frame will be tough!
Because of the way grading works, it is much more important to get the basics right than to get some of the smaller features.
Be sure to think and plan BEFORE you write any code. Having a general strategy about how to tackle this project is the key to doing it. It's too big to swallow whole.
As we have said before, you are permitted to look around for example code and libraries to help you out. However, for this assignment, you must write all of the image processing and compositing code yourself. We expect that you will use libtiff to read and write files, and fltk to make your user interface. You may look at Richard and my sample programs, my ShowChan program may give some good hints as to how to deal with windows nicely. If you are in doubt as to whether using a piece of borrowed code is legal, ask. And remember to give proper attribution!
The basic functionality of a compositing system is to read in a sequence of images and successively combine them together. Fancier compositing systems allow other options.
You will need to support reading in images stored in the TIFF format, applying some image operations to prepare them for compositing, a set of compositing operations (including as over and atop. Preliminary: the exact set you must implement will be specified. An ADVANCED system would implement all the operations in Porter/Duff).
You may assume that the compositing operations are applied in order. E.g. read in image 1, composite image 2 onto that, composite image 3 onto that result, ... You must provide a way for the user to see and/or save the intermediate results. (BONUS: a more clever system may re-order the operations to be faster yet give the same results).
UPDATE 10/10/99: this following paragraph was ambiguous with respect to other things in the document:
NOTE: the first image read in sets the "canvas" that other images are then applied to. The size of the final result (and any intermediate results) will be the size of the first image. Also, you may assume that the canvas is only affected in the rectangular region that it overlaps with the other image.
With the addition of the "canvas" operation (that copies image->canvas), the preferred way to do things is that "read" always sets image, but before any compositing operations happens, a "canvas" operation copies an image to make it the canvas. See the sample script. A system that simply sets the canvas to be the result of the first "read" operation is more limited (if it does not also support the "canvas" copy operation). If you interpretted the instructions as meaning this, you couldn't do the sample script (which was in the originally stated project). All of the test scripts will first read an image and then do a canvas operation, as the sample script does. If your program allows the canvas operation to be omitted, that's cool.
Your program must be able to handle premultiplied and non-premultiplied alpha correctly in image files - both written and read. Use the following method to guess if a TIFF image that you read in is premultiplied alpha or not:
Note: this is a kludge! However, even if you figured out how to get the TIFF "ASSOC_ALPHA" tag correctly from the TIFF file, it is unclear that all the writers do this correctly.
It is up to you as to whether to perform calculations in floating point or integer arithmetic. Ultimately, you will need to convert back to the correct form for displaying and saving.
Your program must allow the user to position where each image is placed. You must handle cases where the image being applied "goes off the edge."
ADVANCED Having a click/drag interface to position images would be nicer than just typing in the position for each image.
Your program must support many of the Porter-Duff compositing algebra operations. These are a little odd since we have a notion of a "canvas" (image A) and an "operand" (image B), so the resulting image is always the size of a canvas. Therefore, we invent some new names so we can always say B op A. Implementing OVER is required, UNDER, IN, and OUT are required. Others are recommended
Remember, we define the operations to occur only over the overlapping area. Other parts of image A are unaffected. Parts of image B that don't cover part of image A will be ignored.
CLARIFICATION: Image B may be "positioned" so that it only partially overlaps A. For example, it might be off the top or bottom.
A lot of this will make more sense when we describe the imaging scripts.
Your program will need to prompt the user for the image files to use in compositing. Preferably, this will be done with a dialog box like the fltk file chooser.
The imaging scripts discussion will provide a more complete list of things to do.
After reading in an image, you must give the user the option of scaling it to the correct size.
BASIC You are only required to support shrinking the image in half and growing the image by a factor of 2 (doubling the size). You do not need to perform proper filtering or any interpolation.ADVANCED Performing reasonable filtering for the shrinking is a good thing to do, and is not too difficult for the fixed half scaling. Similarly, performing interpolation for enlarging images will give you nicer results.
BONUS Scaling by arbitrary amounts can be challenging because it requires proper filtering. Arbitratry, non-uniform scaling is very useful in a compositing system (e.g. fit an image into any rectangle), but difficult to do correctly.
BONUS Other geometric transformations (e.g. non-uniform scales, rotations, ...) are useful in composites.
In your documentation, be sure to indicate what kinds of scaling you support, and the types of interpolation and sampling used to implement it.
You will need to give the user options on how to obtain the alpha channel. Even if the read file contains an alpha channel, the user may wish to create a new one. We will ask you to perform some subset of techniques sch as chroma-keying, blue-screening,
BASIC you must try to do difference matting. Getting it to work well is ADVANCED.BASIC you must do some simple blue screening. Getting it to work well is ADVANCED. You might try using a Vlahos equation, or some simpler version of chromakeying based on color matching. We will try to create examples of varying levels of challenge.
BONUS implement Blinn/Smith Triangulation Blue Screening. If you do, we will try to make images for you. We can't promise anything better than synthetic examples.
BONUS Implement environment matting. That is, use a number of different backgrounds to figure out how light passes through the object, enabling you to show refraction effects. If you’re interested in this, contact us for a copy of a paper describing it. Also, we may not be able to create real test images, but if you get the method to work, we’ll certainly try.
Hint: part of the trick to getting blue screening to work well is to make a good user interface that allows the user to fiddle with the parameters.
A note on examples for this: making images such that blue-screening works well is difficult, requiring carefully controlled lighting and very uniform backgrounds. If you have any interest in photography and would like to help us try to make examples, let us know. We will also create synthetic examples that will be easier to process, and attempt to get some blue-screen shots created by a professional photographer.
Many other image processing operations are extremely useful in a compositing system.
Your program must read script files that tell it what imaging/compositing operations to do. It must also have a user interface that allows the user to do some compositing. However, the two can be the same: you can make your program such that the user simply types the scripts to the program (interactively), and puts up a window when it gets to a show command. Hopefully, you'll make something more interactive that does a point and click user interface - it'll be a lot more fun to use.
When your program is run from the command line, if it is given a command line argument, that will be a text file containing a script.
Having your program write scripts out (based on what the user does) is an ADVANCED feature.
The imaging (or compositing) scripts are plain text files with one command per line. The first word of the line is a verb saying what the line is supposed to do. The rest of the line is the arguments to the verb. The program must be case-insensitive. Your program must stop on an end of file - you can't be sure to get an "end" command.
I will provide you with a basic parser to get you started. We will provide you with sample scripts.
Your program must give reasonable error messages. If you get a verb that you don't understand, give an error message. Note: if a command asks for an advanced feature that your program does not support, your program should say so. It should not give an error, but it should inform the user that it cannot do the command.
Your program always deals with 2 images. A "current" image and the canvas image. Each operation refers to one, the other or both.
CLARIFICATION: The coordinates given are from the upper left of the image (e.g. 0,0 is the upper left corner). Remember that a TIFF image uses a different convention. If you choose to store the image upside-down, then you must interpret the coordinate correctly (remember to display images upside up!)
read background.tif double canvas read bottle.tif bluescreen 5 .5 200 scale .25 showimage composite over 20 20 showcanvas write bottleback.tif
This script reads a background image (background.tif), doubles it (remember background is the "current"), then makes it the canvas. It then reads in a second image (bottle), performs a blue screen extraction, scales the image, then composites the bottle over the canvas. It then writes the image out. The "show" commands have the program put up a window to show the user intermediate results.
You can add additional commands if you wish. For example, if you implement additional operations (like rotate, or blur, or ...).
Here are the commands
You may add others if you want to add extra image processing operations to your system. Anything else not listed here is BONUS.
NOTE: your program must give error messages for commands that you don't implement saying that you didn't implement something. If there is an illegal command, then you program should give a different error message. In general, your program should be robust to problems in the scripts. If something illegal is encountered, your program should give a descriptive error, not crash. You will be penalized if your program crashes.
BASIC The simplest user interface simply allows the user to type in the commands from the scripting language. The show commands would put up a window, and wait until the user closed the window.
ADVANCED A GUI would be a better interface. There is a broad range of possible interfaces. Easier versions would allow a user to choose files with a file chooser, then to perform operations on it, then composite it onto the canvas, and repeat. You might allow a user to position the image by pointing and clicking, provide sliders to experiment with parameters, ...
BONUS A better user interface would allow the user to go back and change what they’ve already done. For example, to reposition an image placed earlier, or even change the order that images are composited into the result. This requires you to keep track of the entire process, and then to re-apply all of the operations.
We will provide a list of questions that you must answer as part of the project. Some of these will require you to create images, some with your system, and some with a painting program such as PaintShop.
We will not give you the questions until a date closer to the due date.
There is a chance that the questions may be an independent homework assignment, with a different due date than the project.
Into your handin directory, you will need to provide the complete source code, project files, and workspace files. You will need to provide a README.txt file explaining what each file you hand in is. You will need to provide a text file called "DOCUMENTATION.TXT" that can serve as a user manual for your program, describing which features you implemented, and how they are accessed.
You may not turn in more than 4 megabytes of files. You shouldn't need to turn in anywhere near this much, but if you feel the urge to turn in lots of big images, be careful. There will be a stiff penalty for violating this rule.
If you do any ADVANCED or BONUS parts, you must say so in your documentation.
You will be required to turn in some example outputs from your program as tif files. Some of these will be the output of running your program on some test scripts.
Part of your grade will be given based on a demo of your system.Scheduling details to follow.
Along with your source files, README.txt file, Questions file (see seperate questions page), DOCUMENTATION.txt, project files (.dsp and .dsw) files, and anything else needed to build your programs (no executable or object files or Visual Studio debugger data!) please turn in:
In your demo you will be asked to do the following: (Hint: try this to practice...)
We will provide you with a number of sample tif images (with and without alpha, for blue-screening, ...) We will also provide a number of test scripts in the compositing file format.
The images will be in the AFS directory: ~cs638-2/public/Images. (that's p:\course\cs638-gleicher\public\Images from NT). There are also some test scripts there, combined with the results of my program.
I will continue to add images as I get them.
Performing matte extraction (by either blue screening or differencing) is difficult in practice.
My intent was not to have you create production quality tools. My hope was that you would gain some experience with the problem and an appreciation for how hard it is. For this reason, I have given you some moderately difficult examples. (also, its hard to create easy examples!).
Getting beyond the basics probably requires some hacking and experimentation. Which I encourage you to do, if you're motivated.
We are not really going to judge the quality of your blue-screening or difference matting as part of your grade. You need to provide something that "works," but we don't expect you to get the quality of results that commercial software does.
By "works," you need to have something that you can demonstrate does actually get right answers on easy problems.
Note: that on the questions and sample images, you will have the opportunity to discuss what you've done.
$header:$