S3 Object Returns True but Doesnt Upload Anything

The problem

In our platforms, nosotros allow our users to upload their own images for profile pictures. This results, as you might imagine, in a wide variety of image sizing, quality and formats. We display these images in diverse ways throughout our platforms.

For the most part, nosotros tin avoid sizing problems by manually setting sizing on the epitome tag. Simply in one very important identify — emails — sure servers ignore our styling and display those images at total size: enormous.

Nosotros need a style to reformat user images programmatically. Additionally, since nosotros're going to be messing with the images anyhow, nosotros'd like to auto-rotate, adjust levels and colors, and by and large make them as nice and as consequent equally we can.

Considerations

• Our images are stored with Amazon'southward S3 cloud storage. Fortunately Amazon offers a relatively piece of cake-to-utilize API for interacting with their services.
• Because our images are on S3, I thought it would exist excellent to have this service as a Lambda office, triggered when a user uploads a photo. Unfortunately I could not, for the damn life of me, get anything to print in the CloudWatch console (where the logs should appear). After bashing upwardly against this wall for a day, I decided to take information technology back in-house.
• We host on Heroku, which offers a gratuitous and simple scheduler to run tasks. Information technology'due south not disquisitional for us to accept these images converted immediately upon upload. Nosotros can schedule a job that picks up everything new in the last 10 minutes, and convert information technology.

The Worker

What's needed at present is a worker we can call every bit oft as Heroku volition allow us (10 minutes is the shortest interval).

Gathering the correct users

First we'll assemble all users that have images that demand to exist converted. We've been storing user images in a specific pattern in our S3 saucepan that includes a files binder. We can merely search for users whose contour pictures Regex matches in files:

                User.where(profilePictureUrl: { '$regex': %r(\/files\/) })              

Your mileage may vary hither, search-wise: we employ a Mongo database.

Of grade, we will be using a different pattern for processed images. This will but selection up those who have uploaded new images since the chore last ran. We'll loop through each of these users and perform the following.

Setting upwards a temporary file

We'll demand somewhere to store the image information we are going to manipulate. We can do that with a tmp folder. Nosotros'll use this equally a holding place for the prototype we want to upload to the new S3 location. We'll name it as nosotros'd similar our terminal paradigm to be named. We wanted to simplify and standardize images in our organisation, so we're using the unique user id every bit the paradigm proper noun:

                @temp_file_location = "./tmp/#{user.id}.png"              

Getting the raw epitome and saving it locally

Now nosotros'll talk to our S3 bucket and get the user's raw, giant, unformatted image:

                key = URI.parse(user.profilePictureUrl).path.gsub(%r(\A\/), '') s3 = Aws::S3::Customer.new response = s3.get_object(bucket: ENV['AWS_BUCKET'], key: central)              

The key code there is taking the URL cord that we've saved as the user'due south profilePictureUrl and chopping off everything that's not the cease path to the picture show.

For instance, http://images.someimages.com/whatever/12345/epitome.png would render any/12345/image.png from that code. That'southward exactly what S3 wants from us to detect the image in our bucket. Hither's the handy aws-sdk gem working for the states with get_object.

Now we can telephone call response.body.read to become a blob of an image (blob is the correct discussion, though it's above my pay grade to really sympathize how images are sent back-and-forth beyond the web). We can write that blob locally in our tmp binder:

                File.open(@temp_file_location, 'wb') { |file| file.write(response.body.read) }              

If we cease here, yous'll see you tin actually open up that file in your temp binder (with the proper name you lot prepare above — in our case <user>.png ).

Process the image

Now we've got the image downloaded from Amazon, we can practise whatever we desire to information technology! ImageMagick is an astonishing tool freely bachelor for everybody.

We used a pared-down version for Rails called MiniMagick. That precious stone also has a dandy API that makes things lickety-dissever easy. Nosotros don't even have to do anything special to option up the image. The @temp_file_location we used before to save the paradigm volition piece of work fine to bring it to MiniMagick'due south attention:

                image = MiniMagick::Prototype.new(@temp_file_location)              

Here's the settings for our photos, but there are tons of options to play with:

                image.combine_options do |img|   img.resize '300x300>'   img.auto_orient   img.auto_level   img.auto_gamma   img.sharpen '0x3'   prototype.format 'png' end              

combine_options is a handy way to practice a agglomeration of stuff to an image in i block. When it exits, the image is saved again where information technology was before. (Image formatting tin't be washed with the img from combine_options.) Now that epitome file in our temporary folder is all kinds of mail service-processed!

Upload back to S3 and salvage as the user's new profile pic

Now all we have to do is set another connection to S3 and make the upload:

                Aws.config.update(   region: ENV['AWS_REGION'],   credentials: Aws::Credentials.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY']))  s3 = Aws::S3::Resources.new proper noun = File.basename(@temp_file_location) bucket = ENV['AWS_BUCKET'] + '-output' obj = s3.bucket(saucepan).object(name) obj.upload_file(@temp_file_location, acl: 'public-read')              

By convention with Lambda, auto tasks will send to a new saucepan with the quondam bucket'due south proper name plus "-output" appended, so I stuck with that. All formatted user images will be dumped into this bucket. Since we are naming the images by (unique) user ids, we are sure nosotros'll never overwrite one user's picture with another.

We create a new object with the new file'south name, in the saucepan of our choice, and then we upload_file. Information technology has to be public-read if we want information technology visible without a lot of headache on our clients (you may cull a different security option).

If that last line returns true (which it volition, if the upload goes smoothly), we tin update our user record:

                new_url = "https://s3.amazonaws.com/#{ENV['AWS_BUCKET']}-output/#{File.basename(@temp_file_location)}" user.update(profilePictureUrl: new_url)              

And that's it! If we run this guy, nosotros'll automobile-format and resize all user images in the system. All the original images volition be in place in their sometime design (and in instance anything goes wrong), and all users' links will point to their new, formatted images.

Testing

We couldn't possibly add a new feature to a Rails awarding without testing, right? Absolutely. Hither'southward what our tests for this look similar:

                RSpec.depict Scripts::StandardizeImages, type: :service do   let!(:user) { User.make!(:student, profilePictureUrl: 'https://s3.amazonaws.com/files/some_picture.jpg') }    before exercise     stub_request(:get, 'https://s3.amazonaws.com/files/some_picture.jpg')       .with(         headers: {           'Accept' => '*/*',           'Take-Encoding' => 'gzip;q=i.0,debunk;q=0.6,identity;q=0.iii',           'Host' => 's3.amazonaws.com',           'User-Agent' => 'Ruby'         }       )       .to_return(status: 200, body: '', headers: {})     allow_any_instance_of(MiniMagick::Paradigm).to receive(:combine_options).and_return(truthful)     allow_any_instance_of(Aws::S3::Object).to receive(:upload_file).and_return(truthful)   stop    draw '.call' do     information technology 'finds all users with not-updated profile pictures, downloads, reformats and and then uploads new picture' exercise       Scripts::StandardizeImages.phone call        expect(user.reload.profilePictureUrl)         .to eq "https://s3.amazonaws.com/#{ENV['AWS_BUCKET']}-output/#{user.to_param}.png"     end   end cease              

If y'all look first at the test itself, you'll see we are testing that our user'southward new profile picture show URL was saved correctly. The residual of it we don't so much intendance about, since we don't really want our test downloading annihilation, and we probably don't want to spend the time for our test to be manipulating images.

But of form the code is going to try to talk to Amazon and spin upward MiniMagick. Instead, nosotros can stub those calls. Just in example this is new for y'all, I'll run through this part.

Stubbing calls

If you aren't mocking calls in your tests, you probably ought to first doing that immediately. All that'due south required is the Webmock gem. You require it in your rails_helper and that'south most information technology.

When your test tries to make a call to an external source, you'll get a message similar this (I've subconscious individual keys and things with …s):

                WebMock::NetConnectNotAllowedError:        Real HTTP connections are disabled. Unregistered request: GET https://... You tin can stub this request with the following snippet: stub_request(:get, "https://...").          with(            headers: {           'Accept'=>'*/*',           'Accept-Encoding'=>'',           'Say-so'=>...}).          to_return(status: 200, torso: "", headers: {})              

Merely copy the stub_request bit and you're well on your fashion to stubbing glory. You may demand to return something in that trunk, depending on what you are doing with the external API call.

I found information technology difficult to get this stubbed response to return something my code would see as an prototype, so I just stubbed the MiniMagick role likewise. This works fine considering we are not seeing the output in this test anyhow. Yous'll take to manually exam that the image is getting the proper formatting.

Alternatively, yous tin use Aws.config[:s3] = { stub_responses: true } in your examination initializer or perhaps on your rails_helper to stub all S3 requests.

I final note: Travis CI

Depending on what options you decide to utilise to your image, you may find that Travis' version of ImageMagick is non the same as yours. I tried lots of things to become Travis using the aforementioned ImageMagick as I was. In the end, I am stubbing the MiniMagick call, so it's a moot signal. Simply beware: if you don't stub that part, you may notice your CI failing because information technology doesn't recognize a newer option (similar intensity).

Cheers for reading!

---

---

Learn to code for free. freeCodeCamp's open source curriculum has helped more than 40,000 people get jobs as developers. Get started

haydenlowat1997.blogspot.com

Source: https://www.freecodecamp.org/news/how-to-post-process-user-images-programmatically-with-rails-amazon-s3-including-testing-c72645536b54/

Share this post