technogeek stuff

What’s Up This Week (er Month) with ISR
It's been too long since I posted an update (there will be a larger email update in April) but here's a quick tour of what's going on.Progress and challenges on the opening sectionThe primary section of that film that is behind schedule - the fly-through of the universe from the Beginning to Earth - about 1/4 of the film - is coming along. We have solved the complex challenges and now it's just a matter of doing the work. We have the unexpected challenge of cleaning up stars out of 5.5 million galaxy cutouts (more about that here). This is going to ultimately delay the film another 4 months, but I'm over halfway through (3 Million+) and working at a faster speed thank to programmable mice and other tricks.The rest of the team working on the opening section has figured out exactly how to place all these 5.5 million galaxies in this section so everything is the correct distance and scale. This will be BY FAR the largest and most complex photographic animation in history. Here's a simple diagram Bill Eberly who leads up the team has put together. He will be horrified since it was just for internal use explaining the complex math and code, but I think it helps give an idea of what exactly we are doing. Click to see the full size.FINAL FOOTAGEFinally, next week I will be in the Outer Banks of NC getting the very last photographs for the film (about 10,000 or so). This was originally supposed to be at the Kennedy Space Center back in April 2014, but if you recall, I suffered a serious ankle injury Feb 24th that year. It was supposed to then be rescheduled for late Dec 2014 when I was recovered enough from ankle and kidney stones, but NASA &KSC red tape has held it up ever since. Finally, I had to cancel and come up with an alternate shoot which I feel is still true to the script and will be just as good. So heading to Rodanthe, NC to get panoramic stills and time lapse at high resolutions to fit the concept of "shores of the cosmic ocean". Before and after packing pics...
What’s Up with ISR This Week
More of the same - and the waiting game. I'm continuing to work on cleaning up rogue stars, corrupt files and problems with the millions of cutouts in SDSS data I mentioned last time. I've developed good speed and getting through a decent chunk every session.Plus, Bill Eberly, lead on the SDSS programming volunteers, wrote code to resize all the cleaned cutouts to 750 x 750 pixels. Not changing the size of them, just added transparent pixels around the smaller ones (the largest ones are around 750 pixels) to insure that we can do scientifically accurate placement in the final renders.So this is what my workspace looks like today (click for high res if you have big monitor) - remote accessing two machines, one manual cleanup of SDSS while running Bill's resize code folder by folder. Occasional errors have to be dealt with manually. It will take about 6 weeks to finish all this. Good to have dual 30" monitors. :)But I can't do this all day, so also continuing to try to schedule to shoot panoramic stills at Kennedy Space Center for the last minute of the film - being working on this for 2 months amidst much red tape and involving, NASA in 4 offices plus SpaceX plus the US Air Force. But it looks like I will finally be able to head there this month. I've revised the ending of the film three times and this ending is the one. Plus work by others on the film on sound, score, Titan imagery tweaking and LRO cleanup.Back to SDSS...
My God, it’s Full of Stars.
http://youtu.be/oALxLNOhI6I?t=1m44s The other reason( than the health issues) that that film is taking longer to complete is the work on the opening section which involves 5.6 million images from the Sloan Digital Sky Survey and thousands of high resolution images from Hubble, ESO and more. We will have a full blog post detailing how this section is being made from the full team, but we are finally solving the uber complex issues and rolling along.Of course, dealing with 5.6 million images there are huge obstacles just sorting, copying etc. Right now I'm getting rid of problematic local star over exposures, bad frames, corrupted files etc. I have several hundred folder of images from 2 to 80,000 to clean out. We are aiming to deal with 100% of visual artifacts that people might actually see and get as close to 100% scientific accuracy as is humanly possible with the photographic data we have.Here are before and after shots of cleaning up a folder.BEFORE:AFTER:So far, about 200,000, only 5.3 million to go :)   ...
What’s Up This Week
Hope everyone enjoyed their holidays. I'm feeling rested and ready to start building my fitness up after nearly 10 months of being unable to exercise.I'm going to post regular updates here on where things are progressing for the film. I will also be putting a more detailed live status page on the website once we figure out the web code. Here's what is happening:Final upgrade to off-site backup setup is nearing completion. This involved upgrading a old Drobo box that had 16 Terabytes of storage and resided in a remote location that due to lack of true high-speed in Greensboro, was too slow and now lacks capacity for backup. That box now has 32 TB and second Drobo unit (we bought both a few years ago at a firesale and they are annoying to work with) was dead but I got it working well enough for offsite backup. Using those old units saved us nearly $5000. It also now has 32TB. Both units have backups of all critical files and critical renders. We also have a local backup of everything (150+ TB) and a cloud backup of critical files (about 10TB) for around 2 complete backups, 3 critical backups and about 6 backups of project files. It took about a month of work just to get this done. But it will be our last upgrade of backups. Team is hard at work on solving the challenges of the opening section that is delayed. This section involves a photographic model of the universe built with data from Sloan Digital Sky Survey (SDSS data). We have completed taking 350,000 photographic fields, cutting out 5.5 million galaxies into individual files and tagging each of those files with distance and location data. That took 9 months. We are now working through the complex math and programming challenges of arranging these 5.5 million photographs of individual galaxies into background photo plates scientifically accurate and into individual layers for those that are close to high resolution photographs processed by our deep space team. Once these challenges are solved we will have a much better idea when the film will actually be done. But it will be worth the wait - it's gonna look amazing. Our local sound mixer Tom Hauser is putting the finishing touches on the "Adagio for Strings" recording and now editing the additional music we recorded at that session. We are waiting final input from the Greensboro Symphony Orchestra conductors on Adagio. A volunteer composer Pieter Schlosser is putting the final touches on the opening 10 minutes of music for the film (that goes along with #2 above). We hope to have other musicians perform this piece for a very unique musical performance up to the task of accompanying the creation of the Universe over 13.7 billion years. I will move on to finishing the cleanup of Colin Legg's timelapse footage that I nearly completed the night before the kidney stones hit as well as working on my tasks in #2 above which includes getting rid of spurious local (local to our Milky Way) in the SDSS data.More soon!...
Major New Milky Way Image Composite
After three years of trying, I finally have the all-sky milky way mosaic photograph needed for the "big shots" in the film - it's 648 Megapixels!!!! This image is to scale showing how much larger it is the one I've used for test shot! Thanks to Axel Mellinger for his amazing work in putting this together.It's created from 3000 original dark sky photographs in Texas, Michigan and my home country, South Africa. Amazing work and will be spectacular in the film.Read more about it at Axel's website....
Adventures in Data Storage, Part 2
PART 1 is here.As I was saying, the problem began when a routine file copy locked up the main Frankenserver - which should not happen. Saturday morning, I checked and found a BIOS update for the motherboard in the Frankenserver was available. Normally, a BIOS update is very routine. This was not.After applying the BIOS update, the main storage array (16 Terabytes) was corrupted and inaccessible. Not a great feeling. But I was not panicked yet as even if you delete files on hard drives, the files are there until they are overwritten with new data. So I knew the data was just there as the drives all appeared just fine.But I needed a new motherboard as this one appear to have had an electrical failure. On a holiday weekend, my only solution was drive to a nearby (an hour and 45 minute drive) city to get a part before it closed. So I drove, got the board, drove back, ate, then installed the new board...
Adventures in Data Storage, Part 1
Back posting again after a holiday weekend here in the USA as well as an adventure to recover damaged data. On Saturday morning, I wanted to check up on a hard crash that had occurred on the main "Frankenserver" that stores a backup of all the Outside In and SV2 Studios files.Here's a pic of secondary Frankenserverbeing upgraded last year - I call them that as they are built from the parts of other machines to recycle parts and keep the film's costs as low as possible.One question I get a lot is how I store and backup data. I've made a simple diagram that gives an overview of the design and process. This diagram is simplified but all the key steps are there. I have two primary workstations - the primary is the fastest, most powerful and has a large, fast disk array for photographic storage and the secondary is used when the primary is tied up and also for freelance work as needed.I have 3 other workstations - an audio/recording workstation and secondary render box plus two low-end boxes: a Linux machine and a Hackintosh. Then there are two Frankenservers, the primary that has 27 TB of storage and is connected to a cloud backup and the secondary provides additional utility storage (workstation backup etc.).The primary Frankenserver is connected to cloud backup storage (CrashPlan). The overall design of this is to keep the costs of the film 1/10th of what it would normally but create a system that can recover, eventually, from even a catastrophic data loss - say an alien warship destroying my house (but not destroying earth, but then again, that will not be a big worry at that point).This design also allows me to work on the film on multiple stations although the primary workstation is fastest for the heavy lifting. It consists of low-cost, DIY, home-built computers overclocked to provide near professional workstation speeds at 1/4 of the cost. And the storage and backup costs are a tiny fraction since it relies on the very cheapest hard drives and cloud storage that costs only $5 a month combined with a $20 month upgrade on internet service for fast upload speeds.The bottom line is all critical files are backed up in at least two locations and 3 to 6 copies of each file. Storage costs on the film are currently 4 cents a gigabyte and offline backup costs are less than 1 cent a gigabyte. Currently I have about 6 Terabytes backed up offline but the number grows everyday.It's a good design especially given the cost - the only disadvantage is a slow restore process in the event of large data loss but this past weekend, it was tested.PART TWO TOMORROW - what happens when the main server motherboard dies and corrupts the main data storage array....
EXPOSED! Outside In’s dirty little secret (and hot juicy details about Adobe After Effects CS5)
Shocking headlines seems to work for the rest of the media, so why not here :) I've been sitting on a "dirty little secret" about Outside In for almost exactly a year. But now, thanks to the release of Adobe CS5, the ugly truth (and I mean ugly) can come out. I have been under an NDA (non-disclosure agreement) and had to sit on the good news for a long time.The frightening secret (well, it was frightening to me) - Outside In could not be made as described, as promised. Severe memory problems and limitations in previous versions of Adobe After Effects had left the film with some ugly choices. How ugly? Well pictures can tell a story.If you read these posts, you'll recall that IMAX is a huge canvas. You need images that look good at huge sizes on an 8-story high screen. We've all experienced trying to blow up a picture and the fugly results. Here's what happens to a nice image of Saturn available online:Now, let's say you want to make it IMAX size and zoom then flyby the image:Okay, that's not pretty. But you say, well, maybe I can live with that (actually I can't but some people are okay with uprezzing). But the real crime is about light. Light is the miracle of the universe and the prime ingredient in art and of course filmmaking. Those wonderful little photons, emitted by the sun, traveling billions of kilometers to hit Saturn, bounce into the Cassini camera and then transmitting back to earth to be projected in a beam of light onto the screen that reflects those photos into your eyes and into your mind. Wow. Mind blown.But that amazing journey has a cost. Especially when you start processing images on your computer. Most all computer image processing is done is something called "8-bit color". Basically that means that there are 256 values assigned for light and darkness. Well, that does make it easy to work with on computers, especially in years past, but it's huge violation of the natural world. Our eyes see more than this of course, but the real issue is what happens when you heavily process an 8-bit image. Bad things, bad, bad things.So I knew that for this to work, I needed to be at least in 16-bit color - which gives you over 16,000 values for light and dark. The first couple of film-out tests for Outside In were done at 4000 x 3000 pixels in 16-bit color. They looked good but I still had to hide the dirty little secret. The flyby of Saturn rings? My recipe for creating it required intensive use of processing light in Adobe After Effects in order to main the visual integrity of the photographs. This is what the process looks like at various color depths.First, 8-bit color, the standard for digital cameras, the web, desktop computing, video etc. :Now the exact same thing but in 16-bit color (some pro photo and video people work in 10, 12 or 14-bit and process in 16 bits, but it can be difficult and cumbersome):And now, 32-bit color, as nature intended and Adobe After Effects CS5 makes possible at very large frame sizes with it's new 64-bit engine:So, you can see - the dirty secret on Outside In was the film could only be made at high resolution in low-color or low resolution in high-color. But not both. So last year, after outputting shots without the super coolness, I began looking at options for moving to another software program. But the results of my search would not encouraging (and often would require spending the price of a new house to find out).Then I heard a rumor of a new version of After Effects. I decided to wait - I helped my friend Jason Pierce shoot his film "Hellphone" and worked on other projects. Then I was lucky enough to have access to a prerelease version of the new After Effects. You can imagine the very, very, very first thing I tried to do with it. And yes it immediately worked and was near ballin' my eyes out. But I could say nothing, until now.But it's on - Outside In is in production. Feel free to sign up with Team 11 and help bring these stunning images to the giant screen....
11 Reasons to NOT to make an IMAX art film in your basement: #7: “Too Much Data”
Computers in the future may weigh no more than 1.5 tons. Popular Mechanics (1949)#7: “Too Much Data” If you recall from reason #5, "a 5.6k IMAX image at 32-bit color will contain 752,640,000 bits of data or nearly 1 Gb (gigabyte) for each frame." In non-technobabble that means the finished film will require 65 terabytes of data - which is 65,000 Gigabytes of data. That does not count the many terabytes of data for all the source material in the film. 100 terabytes in all.In 2006, the 750 gigabyte hard drive was just released, for over $500. To store 100 terabytes would have taken 133 hard drives costing $67,000 dollars. So that was near deal breaker when "Outside In" was first conceived.Today, we have 2 terabyte hard drives for $135. That's 50 drives costing $6,750.  So that's getting near indie low/no budget territory right? Indeed it is, especially since you can buy storage as you go along. But there is more to just storing files, you have to make them first with gobs of data.After a few initial tests that went very well, I started tying to do animation on "Outside In". That 1 GB of data per frame? Well, your computer has to process that data. And film is 24 frames per second - 24 gigabytes of data per second. Things like this started to happen:The first time you see this I thought - "okay, just another error message, no problem. Just find the fix or work around". Several months later, after talking to the company, many expert users and lots of research I realized with a sickening feeling in my stomach that it's not just another error message.Turns out it's a fundamental problem with computing.  In 2006/7, pretty  much every operating system and programs were "32-bit". Which means basically the same computing technology based that had existing for well over a decade or two. Yes, some hardware like the processor was "64-bit" but it was basically sitting there doing very little 64-bit as all the code it ran was 32-bit.Why is that such a problem? Well 32-bit code basically can only deal easily with about 2 gigabytes of data - and tweaks had allowed to deal with 3 or 4 GB of data but just barely. So that 1 GB of data per frame of IMAX data (that often had many gigabytes of data placed inside the frame) was simply too much. Code crashed, operating system crash - and film crashed.But I was determined to press ahead. A few months later, Microsoft finally released a 64-bit version of Windows XP.I installed it on a dedicated machine as it was very picky about what hardware and software I could run. I got Adobe Photoshop and Adobe AfterEffects running and despite that fact the both were still 32-bit code, running them on 64-bit XP made things better. I could do a lesser resolution that my original plan (4k instead of 5.6k) and less colors (16-bit not 32-bit) but it was good enough to create some serious test shots. My first few minutes of footage were output to IMAX film and screened in an IMAX theater. Despite the compromises, it looked much better than I could have hoped.But, as 2008 drug on into 2009, I was faced with a fundamental crisis. The film, as scripted and storyboarded, required much more data than I could animate with the workarounds around the 32-bit memory limitations. I could either make major changes to the film - and make a much simpler, less dynamic, less visually interesting and rich film. Or I could try to raise a lot more money and move to Hollywood level animation system - and from what I could find out, none of them were true 64-bit either and had similar issues.Neither of these seemed like good ideas. I took a hard look at the future and felt that I was only 12-18 months away from a real revolution in desktop computing - the move to true 64-bit computing. But I did not know for sure if that would save my vision for the film....