CSE 457 Introduction to Computer Graphics
Winter Quarter 2015

Project 1 : Impressionist

Assigned: Thursday, January 8th

Due: Thursday, January 22nd by 11:59 PM

Artifact Due: Monday, January 26th by 8:00 PM

Help Sessions:
Monday, January 12th at 4:30 PM in CSE 203
Wednesday, January 14th at 4:30 PM in CSE 403

Project TA: Alon Milchgrub

Quick Links

Impressionist Details

Working From Home / Installing FLTK

Your computer needs the Fast Light ToolKit library to compile class projects:

NOTE: all projects must be submitted as Visual Studio 2013 projects with binaries.  You may use another OS for development, but the project must ultimately compile under VS 2013 and run on Windows. 

FLTK and OpenGL Resources

Sample input images and examples

Project Description

Impressionist is an interactive program that creates pictures that look like impressionistic paintings. It is based on a paper and a program by Paul Haeberli. Here is a copy of his paper "Paint by Numbers".

To create an impressionistic picture, the user loads an existing image and paints a seqence of "brush strokes" onto a blank pixel canvas. These brush strokes pick up color from the original image, giving the look of a painting.  See some examples of what you can do with Impressionist, from previous quarters: examples, examples, examples.

This is motivated by impressionist paintings that artists have been making for centuries. To help inspire you when creating new brush types, or when creating your artifacts, here are the Google Images "Impressionist Art" results.

Project Objective

You will add the functionality to a skeleton version of the Impressionist program, which we will provide. The purpose of this project is to give you experience working with image manipulation, OpenGL primitives, user-interface design, and image processing.

Getting Started

To get going, you need to get the skeleton source code. This is distributed via SVN, which is all set up for you. In the labs, we will be using TortoiseSVN. In order to get the source code, follow the directions below:

Graphics Lab:

  1. Create a new folder somewhere (such as your Z: drive) with the name 'impressionist', right click on it, and choose 'SVN Checkout'
  2. In the 'URL of repository' field, enter:
    file://\\ntdfs/cs/unix/projects/instr/15wi/cse457/impressionist/YOUR_ASSIGNED_GROUP_NAME/source
    When copying this string (then editing the group name), be careful that the "\"s and "/"s are preserved exactly as they appear on this page.
  3. Make sure that the 'HEAD revision' radio button is selected, and hit OK
  4. A working copy of the source code will be checked out into the folder. You're now ready to work!

Remote Access:

If you plan to work from home, you will need to point Visual Studio to the correct include and library directories for the FLTK that came in your repository (since normally it points to the local one on each lab machine). Instructions for how to do this. To open and build the project, double-click impressionist.vcxproj.

Note: If you are having errors in the #include lines of any of your header files, copy over the fltk1.3.2 folder from C:\Program Files (x86)\ into C:\Program Files. This is an issue where the build is looking in the wrong folder.

Explanation of the Skeleton Program

The skeleton program we provide does very little. It allows you to load the original image (which can be in BMP, PNG, or JPEG format), and save the painted version. Brush selection is done via a drop down list on a separate window called up via the "File" menu. There is one brush implemented (points) and a slider for controlling the brush size.

Here are some sample BMP images.

Required Extensions

You must add the following features to the Impressionist program:

  1. Implement 5 different brush types: single line, scattered lines, scattered points, (filled) circles, and scattered (filled) circles. See the sample solution for an example of each brush's appearance. Note that scattered brushes should sample from each location they color individually, not just use a single color for each splotch.
     
  2. Add sliders to control various line brush attributes. You need to include sliders for the thickness (width) and orientation (angle) of line brushes, in addition to the existing brush size slider.
     
  3. Add the ability to control the line brush orientation with the image gradients. The line brush orientation should be controlled two different ways: using a slider value (see #2 above) and using directions that are perpendicular to the gradient of the image. You can use a radio box to allow the user to select which method to use.

    Note:

    • Use the Sobel operator to determine the gradient.
    • We recommend using the atan2(y,x) function instead of atan(y/x), because it handles the case where x=0. Note that its return value is between -pi and pi.

     

  4. Allow the user to change the opacity (alpha value) of the brush stroke. An alpha value slider should be added to the controls window. You may want to consult the blending subsection of the OpenGL Red Book for this. (Pages 162-169 of the linked PDF are particularly helpful.)
     
  5. Implement the filter kernel. The skeleton code already provides the user interface. You should be able to specify any 5x5 filter by typing in the filter coefficients, a scale factor by which each filter coefficient is divided, and an offset which is added to the pixel before displaying. The filter is applied to the entire "Paintview" image (the image on the right panel).

    Note:

    • You will need to implement a method for handling boundary pixels, when part of the filter kernel goes off the edge of the image. You must do something "smarter" than assuming that the image is black beyond its boundary (i.e., do something other than zero-padding).
    • The filter kernel allows users to enter values such that the resulting pixel value is out of the range [0...255] or the kernel values are divided by 0. You will need to do something "reasonable" to handle these cases.
    • The filter is applied to the painted image (instead of the original image). Also, the filter kernel dialog is a modal dialog. You must first close the filter kernel dialog before you can continue painting on the "Paintview" canvas. Also, be advised that OpenGL stores the image in memory upside-down - i.e if the image is a 3 x 3 grid with [1 2 3] as the top row followed by [4 5 6] and [7 8 9] as the other two rows, then the array will be [7 8 9 4 5 6 1 2 3]. You will need to compensate for this when implementing the filter kernel.
  6. Implement at least one bell's worth from the list of bells and whistles (Note: 1 bell equals 2 whistles). Any additional bells and whistles will be extra credit.

To see what these features should look like when they're done, you can look at the sample solution (with some of the extra credit). Your implementations of brush strokes, brush direction controls, etc. do not have to behave exactly the same as the sample solution, but they should be fairly close.

(Note: the "Edge Image" extra credit button implemented in the Sample Solution skews the image. This is NOT correct, but remember - it's extra credit.)

Project Turnin

Instructions Here

Project Artifact

When you are done with this project, you and your partner will each create a project "artifact" to show off the features of your program.

For the Impressionist artifact, you will create an impressionistic painting from an image of your choice.
We will then create a gallery of all the paintings on the course web page where you will vote on your favorite artifacts!!

For these artifacts, please turn in the original and impressionized version in jpg or png form.

Artifact Turnin Page

Bells and Whistles

Here is a list of suggestions for extending the program for extra credit. You are encouraged to come up with your own extensions. We're always interested in seeing new, unanticipated ways to use this program!

whistle To give your paintings more variety, add some additional brush types to the program. These brush strokes should be substantially different from those you are required to implement. You will get one whistle for each new brush (within reason).

whistle The skeleton program allows the user to paint outside the boundary of the paint rectangle, then erases this region when the stroke is completed. Change this to clip brush strokes to the region as they're being painted.

whistle When using your program, you currently can't see what part of the original image you're painting. Extend the program so that when the cursor is in the painting window, a marker appears on the original image showing where you're painting.

whistle Sometimes it is useful to use the contents of the painting window as the original image. Add a control to swap the contents of the painting window and the contents of the original image window.

whistle For your filter kernel, add a "normalize" checkbox that will automatically divide by the sum of the weights when the user wishes it.

whistle Add the ability to control the line-brush stroke direction and size using the right mouse button to drag out a direction line.

whistle Add the ability to control the line-brush stroke direction using the direction of the cursor's movement.
(Hint: The immediate brush directions obtained from mouse movement can be quite noisy, and it is nicer if they are smoothed out with respect to previous brush directions. You will need to handle this issue. Note that, if you remember the brush directions as a list of angles, you cannot just average the angles. A simple average doesn't account for the fact that two brush angles, such as 359 degrees and 0 degrees, may be closer than their numeric values imply.)

bell Add controls that allow you to manipulate the color of the image. For example, you could implement independent scaling of the red, green, and blue channels.

bell Design a brush that selectively applies one or more filters from your filter kernel. This might require some UI changes to your filter kernel UI. Note: you must take into account the brush size. picture

bell Add an undo feature with at least one level of undo so that you can try a brush and decide to undo its effect on the canvas. This comes in very handy for experimenting with brush and filtering effects.

bell Add the ability to dissolve one image into another.

bell Fix the Red Line Bug in right mouse drag out direction line. It is possible for a series of odd mouse clicks during a right mouse drag to permanently draw the red line on the canvas (spoiling a possibly beautiful drawing). Eliminate this problem entirely by properly synchronizing mouse events.

bell+whistle A different solution to the problem of not being able to see where you're painting is to show a dimmed version of the painting on the canvas. Add a slider that allows the user to fade in or fade out the original image beneath the user's brush strokes on the canvas. (Beware, this bell and whistle is more difficult than it looks).

bell+whistle Add a "mural" effect to your Impressionist by implementing the ability to load in different images while preserving what has been drawn on the canvas. Add a "New Mural Image" or "Change Mural Image" to the controls window that allows the user to change images. The user may then load an image, draw in what he / she prefers on the canvas, and then load a different image and continue drawing on the canvas; thus, a "mural" effect.
picture

bell+whistle To make your painting more interesting, add "alpha-mapped" brush strokes. In other words, allow the user to load a bitmap representing a brush stroke. This bitmap would contain an alpha value at each position. Then when this brush is used to draw, a single color would be selected from the image, all pixels in the brush bitmap would be set to this RGB color (without changing the alpha value), and this partially transparent bitmap would be painted on the canvas. A new color would be used each time the brush is drawn.

bell+whistle It can be time-consuming to paint an image manually. Add a feature so that a whole painting can be created automatically. The user should only have to specify a brush type, size, and angle to use. Then the program should automatically paint brush strokes over the entire image, using a randomized brush order and varying the brush attributes slightly as it goes (to increase realism).

bell+whistle At times, you may want the brush strokes to follow the gradient of a different image than the base image. Add a button(s) that will cause the direction of brush strokes to be automatically determined from a user specified image.

bell+whistle The "accuracy" of the painting can also be improved by clipping long brush strokes to edges in the image. Allow the user to load a black-and-white image that represents the edges in the picture. Then add a checkbox so that the user can turn on edge-clipping, which will automatically clip brush strokes at edges in the image.

bellbell Use the image processing techniques described in class to automatically find the edges in the base image. Once you have found the edges, add a button to the user interface that will allow the user to select whether or not the brush strokes should be clipped to the edges in the picture.

bellbell Implement a curved brush that follows the image gradient. See Painterly Styles for Expressive Rendering.
picture

bellbell Implement a bilateral filter. You should provide sliders that control how wide the filter is: one slider for the domain width, a second slider for the range width. You earn one bell for implementing this filter just as a specialized mean filter, where you simply average values that are within the domain/range widths of each image pixel. For an extra bell, use Gaussian filters, and add a toggle to switch between bilateral mean and Gaussian filtering.

bellbellbell Implement a multiresolution automatic painting technique. See Painterly Styles for Expressive Rendering.

bellbellbell Design a brush that can be used to stretch and pull the image as if it were rubber.
paper link

bellbellbellImplement "animated" brush strokes that make the image appear to move in interesting ways. Ex. You could paint moving ripples over a picture of a lake, or rustling motions onto grass or tree. Credit will vary depending on the success of your method.

bellbellbell Given a source image, construct a new image that is really a mosaic of small (thumbnail) images. To do this, you need to partition the original into tiles and find new thumbnails that are reasonable matches to the tiles. Then draw the new image by substituting the thumbnails for the tiles. See, for example, Adam Finkelsteins Web Gothic. Mosaic sample solution (stand-alone command line executable). An image that was created using the sample solution. Credit will vary depending on the success of your method. To get full credit, you must perform some sort of edge detection to accurately determine which thumbnails to use, and you must use the original color of the selected thumbnails.
picture

Disclaimer: please consult the course staff before spending any serious time on these. These are all quite difficult (I would say monstrous) and may qualify as impossible to finish in the given time. But they're cool.

Impressionist Video (8+ bells)

bell bell bell bell monster bell bell bell bell bell

Implement a method to automatically create non-photorealistic video. One very simple method (that would not get a monster bell) would be to run auto-paint on each frame of a sequence. For credit, your technique should exhibit temporal coherence.

Other artistic methods, such as charcoal sketch, often de-emphasize the background (or leave it out altogheter). When processing still images, it is practically impossible to distinguish the subject and the background without any human assistance; however, in a video stream, it may be possible to exploit movement to segment the image. For additional credit, implement a method that effectively uses this method to generate a convincing non-photorealistic version of live video.

For even more extra credit (and probably a conference paper) do all of this in real-time on a consumer PC.

Image collages (8+ bells)

bell bell bell bell monster bell bell bell bell bell

Image mosaics are often pieced together by stitching together a bunch of tiny rectangular images. Although this produces a cool effect, it looks computer generated. Implement a method to build collages, given a sample set of images. The primary difference is that the shapes need not be rectangular and that they can also overlap. A while back, some graduate students here implemented a method to do this, ultimately resulting in building a face with pictures of fruit.

Another approach is to note that, when humans build collages, we usually clip shapes out of images (cutting out a picture of a red car and pasting it in as someone's upper lip, for instance). Given a set of data images, we wish to automatically build a collage of some input image, given that we can cut simple shapes from the data images. If you've seen The Truman Show, you may remember that Truman puts together a picture of a woman's face using magazine clippings. This took him a while.

Here's a paper from Siggraph 2002 Jigsaw Image Mosaics

Non-videorealistic transformation (8+ bells)

bell bell bell bell monster bell bell bell bell bell

Okay, I completely made up that term. In artistic animations, the movement is often not completely realistic. One technique that has been used for some advertisements and music videos involves sampling the video at a very slow frame rate (say, two per second) and then filling in the disarded frames using morphing. You may want to use the optical flow (see CSE490CV) to assist with the morph. You may also want to split up the image, morphing different regions and varying frame rate according to how much movement there is. After you perform this pass, maybe try running your impressionist program on each frame, using the morph and the optical flow to guide the direction of temporally coherent brush strokes.

For even more extra credit (and probably a conference paper) do all of this in real-time on a consumer PC.

Design by Ian Li.
Winter 2015 maintenance by Xiangguaneg Zheng & Bobby Gebert (contact info on personnel page)
CSE 457 Introduction to Computer Graphics
Winter Quarter 2015
 
 
  Last modified: Tuesday, 13-Jan-2015 18:39:31 PST