Category Archives: amazon-s3

Ubuntu server shows memory error

I have an EC2 instance with Ubuntu 16.4 for running my Codeigniter website.This website is getting slow and shows insufficient memory time in some times.this is not on a specific time period and the server was down and the website was unable to access on that time. I have checked the log and it shows some non existing files are trying to access from the server.Here is the part of the error log.if some one have any idea about this.please give some clue

fcntl(15, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = 0 epoll_wait(16, {{EPOLLIN, {u32=3392909944, u64=140277024797304}}}, 2, 10000) = 1 accept4(6, {sa_family=AF_INET6, sin6_port=htons(62318), inet_pton(AF_INET6, "::ffff:195.11.198.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28], SOCK_CLOEXEC) = 17 fcntl(15, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0 gettimeofday({1502802731, 119862}, NULL) = 0 getsockname(17, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "::ffff:172.31.28.156", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0 open("/dev/urandom", O_RDONLY|O_CLOEXEC) = 19 read(19, "Mq\7\273\272\341e\322\340\354&\261!\336\267\236\\354\251C\332\376r\262\301pEP\301\242\350\330"..., 512) = 512 close(19) = 0 fcntl(17, F_GETFL) = 0x2 (flags O_RDWR) fcntl(17, F_SETFL, O_RDWR|O_NONBLOCK) = 0 gettimeofday({1502802731, 120442}, NULL) = 0 read(17, "\26\3\1\2\0\1\0\1\374\3\3\263\217T\301\315\37o\324\320\322H\207t\254^[email protected]\260\331\364\302"..., 8000) = 517 writev(17, [{"\25\3\3\0\2\1p\26\3\3\0Q\2\0\0M\3\3\247\320\371\215\373Ie_\216)va\2133"..., 144}], 1) = 144 read(17, 0x7f94ca3b0048, 8000) = -1 EAGAIN (Resource temporarily unavailable) poll([{fd=17, events=POLLIN}], 1, 300000) = 1 ([{fd=17, revents=POLLIN}]) read(17, "\24\3\3\0\1\1\26\3\3\0(\0\0\0\0\0\0\0\0\265\221xP\26\240Z\20\245\211\t\f\347"..., 8000) = 51 poll([{fd=17, events=POLLIN}], 1, 300000) = 1 ([{fd=17, revents=POLLIN}]) read(17, "\27\3\3\4^\0\0\0\0\0\0\0\1g3\257Q\377\307 \322\210b\232\f\355\[email protected]\4\356\274\37"..., 8000) = 1123 gettimeofday({1502802731, 208274}, NULL) = 0 gettimeofday({1502802731, 208368}, NULL) = 0 gettimeofday({1502802731, 208401}, NULL) = 0 stat("/var/www/html/index.php/backend/members/add_member", 0x7fff525da000) = -1 ENOTDIR (Not a directory) lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lstat("/var/www", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 open("/var/www/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) lstat("/var/www/html", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0 open("/var/www/html/.htaccess", O_RDONLY|O_CLOEXEC) = 19 fstat(19, {st_mode=S_IFREG|0664, st_size=288, ...}) = 0 read(19, "RewriteEngine On\nRewriteCond %{H"..., 4096) = 288 read(19, "", 4096) = 0 close(19) = 0 lstat("/var/www/html/index.php", {st_mode=S_IFREG|0664, st_size=10252, ...}) = 0 gettimeofday({1502802731, 208815}, NULL) = 0 gettimeofday({1502802731, 208838}, NULL) = 0 gettimeofday({1502802731, 208919}, NULL) = 0 stat("/var/www/html/backend/members/add_member", 0x7fff525d9da0) = -1 ENOENT (No such file or directory) lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lstat("/var/www", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 lstat("/var/www/html", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0 lstat("/var/www/html/backend", 0x7fff525d9da0) = -1 ENOENT (No such file or directory) setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={60, 0}}, NULL) = 0 rt_sigaction(SIGPROF, {0x7f94c5fc74b0, [PROF], SA_RESTORER|SA_RESTART, 0x7f94c96b0cb0}, {0x7f94c5fc74b0, [PROF], SA_RESTORER|SA_RESTART, 0x7f94c96b0cb0}, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0 getcwd("/", 4095) = 2 chdir("/var/www/html") = 0 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={3000, 0}}, NULL) = 0 fcntl(12, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1, len=1}) = 0 getcwd("/var/www/html", 4096) = 14 stat("/var/www/html/system/", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 stat("application", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0 getcwd("/var/www/html", 4096) = 14 stat("/var/www/html/application/views/", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 access("/var/www/html/application/config/production/constants.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/config/config.php", F_OK) = 0 access("/var/www/html/application/config/production/config.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/core/Benchmark.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/system/core/Benchmark.php", F_OK) = 0 access("/var/www/html/application/core/MY_Benchmark.php", F_OK) = -1 ENOENT (No such file or directory) gettimeofday({1502802731, 210780}, NULL) = 0 gettimeofday({1502802731, 210807}, NULL) = 0 access("/var/www/html/application/core/Hooks.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/system/core/Hooks.php", F_OK) = 0 access("/var/www/html/application/core/MY_Hooks.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/core/Config.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/system/core/Config.php", F_OK) = 0 access("/var/www/html/application/core/MY_Config.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/core/Log.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/system/core/Log.php", F_OK) = 0 access("/var/www/html/application/core/MY_Log.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/logs/", F_OK) = 0 stat("/var/www/html/application/logs/", {st_mode=S_IFDIR|0775, st_size=12288, ...}) = 0 access("/var/www/html/application/logs/", W_OK) = -1 EACCES (Permission denied) access("/var/www/html/application/core/Utf8.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/system/core/Utf8.php", F_OK) = 0 access("/var/www/html/application/core/MY_Utf8.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/core/URI.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/system/core/URI.php", F_OK) = 0 access("/var/www/html/application/core/MY_URI.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/core/Router.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/system/core/Router.php", F_OK) = 0 access("/var/www/html/application/core/MY_Router.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/config/routes.php", F_OK) = 0 access("/var/www/html/application/config/production/routes.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/controllers/Backend.php", F_OK) = -1 ENOENT (No such file or directory) stat("/var/www/html/application/controllers/backend", {st_mode=S_IFDIR|0775, st_size=24576, ...}) = 0 access("/var/www/html/application/controllers/backend/Members.php", F_OK) = 0 access("/var/www/html/application/core/Output.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/system/core/Output.php", F_OK) = 0 access("/var/www/html/application/core/MY_Output.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/config/production/mimes.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/config/mimes.php", F_OK) = 0 access("/var/www/html/application/cache/4811d3d32f806d8a708366b1d7ecf8a7", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/core/Security.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/system/core/Security.php", F_OK) = 0 access("/var/www/html/application/core/MY_Security.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/core/Input.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/system/core/Input.php", F_OK) = 0 access("/var/www/html/application/core/MY_Input.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/core/Lang.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/system/core/Lang.php", F_OK) = 0 access("/var/www/html/application/core/MY_Lang.php", F_OK) = -1 ENOENT (No such file or directory) access("/var/www/html/application/core/MY_Controller.php", F_OK) = -1 ENOENT (No such file or directory) gettimeofday({1502802731, 213685}, NULL) = 0 access("/var/www/html/application/controllers/backend/Members.php", F_OK) = 0 gettimeofday({1502802731, 213879}, NULL) = 0

How can I host my website on an EC2 Server using Apache but host my images on S3 under the same sub domain?

So I have my website hosted on an EC2 server running apache. All works fine and well.

The issue I'm having is that I want my images hosted on S3 (and possibly cached on CloudFront), not on my EC2 server, but want both accessible under the same domain/subdomain.

For example say my website is www.helloworld.com. I want my images to be accessible at www.helloworld.com/images/foobar.png. I want foobar.png to be hosted on S3 and not each one of my EC2 servers though. How can I go about doing this?

Is there and configuration I can do in the AWS console or are there any rewrite rules I can use in Apache? I know I could always just set up a controller that will download the images from S3 and forward them to the user, but this seems wrong to me.

how to authenticate the S3 url using httpd pass_proxy

My requirement is to authenticate the S3 objects url using httpd. As I am using the url redirection using httpd using the mod_proxy. I need the S3 url should be authenticated using the httpd.

Ex: /abc.jpg https://s3-region.amazonaws.com/bucket-name/abc.jpg

The above S3 url needs to be authenticated such that I should be accessed locally with any access denied.

Thanks

Nitesh

how to route www.domain.com to httpswww.domain.com in Route 53?

We are trying to route www.example.com to https://www.example.com using route 53 when ever user search for www.example.com

We redirected all the traffic from example.com to https://www.example.com by using S3 bucket Redirect all request, then assigned alias target as example.com bucket in route53.

Now the challange we are unable to do the same on www.example.com wherin loadbalancer DNS is linked with www.example.com as alias target in Route53. So how do i can route the traffic from www.example.com to https://www.example.com

Thanks in advance for your help!!

Apache zeppelin unable to fetch URL from Amazon S3 notebook json file

Hi I am unable to import a notebook from amazon S3 after following tutorials and instructions on AWS site. Specifically, 1. I have a notebook created by Zeppelin in S3, 2. I have spun up a cluster with Zepplin, Spark, etc. installed 3. I provided a JSON configuration file on creating the EC2 clusters so that Zeppelin would save files to my S3 bucket:

[ { "Classification": "zeppelin-env", "Properties": {

},
"Configurations": [
  {
    "Classification": "export",
    "Properties": {
    "ZEPPELIN_NOTEBOOK_STORAGE":"org.apache.zeppelin.notebook.repo.S3NotebookRepo",
      "ZEPPELIN_NOTEBOOK_S3_BUCKET":"bucket-name",
      "ZEPPELIN_NOTEBOOK_USER":"user-name"
    },
    "Configurations": [

    ]
  }
]

} ]

  1. When I try to import a note on Zeppelin using the note URL provided by Amazon I get the error "Unable to fetch URL".

Who carries the bandwidth burden of serving content in redirect an image file to another domain in .htaccess

I put my image in AWS S3 bucket

https://s3-us-west-2.amazonaws.com/mybucket/img1.jpg 

I don't know whether it is the most efficient way to do it or not, but I use

https://www.example.com/images-s3/img1.jpg 

in my html content and I use redirect rule in .htaccess to redirect the image to AWS S3 bucket.

RewriteRule ^/?images-s3/(.*)?$   https://s3-us-west-2.amazonaws.com/mybucket/$1 [L,R=301]

I am wondering in this case does my server also carry the bandwidth burden to deliver the image as well besides my AWS S3 bucket ? That is to say - if I do it this way, do I need to pay for the bandwidth of delivering the image both in AWS S3 and in my server on AWS EC2 ?

Is there another way to do it that can save me cost ?
Can I do that in a better way using AWS Route 53 ?

how to forward request from apache to s3

I have a hosted website on fasthost DNS provider with the name mywebsite.com. Any request to mywebsite.com gets forwarded to a running on aws EC2 instance on which my web application is running. In the EC2 instance, I have setup Apache http server that takes all the requests and forward to my web application with the help of the following virtual host setting in the apache httpd.conf file

<VirtualHost *:80>
     ProxyPreserveHost On
     ProxyRequests Off
     ServerName mywebsite.com
     ProxyPass / http://127.0.0.1:8080/
     ProxyPassReverse / http://127.0.0.1:8080/
</VirtualHost> 

So far this is working fine. Now I have static image which I have hosted on aws S3 bucket. The publicly accessible url is https://s3.eu-west-2.amazonaws.com/static/images/some-img.png

I do not want the S3 domain-name appear and would like to mask s3.eu-west-2.amazonaws.com server name so instead of https://s3.eu-west-2.amazonaws.com/static/images/some-img.png I have something like https://mywebsite.com/static/images/some-img.png I tried adding the Virtual host entry in httpd.conf so that any request with context /static gets forwarded to S3 instance. but it doesn't seem to be working:

<VirtualHost *:80>
     ProxyPreserveHost On
     ProxyRequests Off
     ServerName mywebsite.com
     ProxyPass /static https://s3.eu-west-2.amazonaws.com/
     ProxyPassReverse /static https://s3.eu-west-2.amazonaws.com/
</VirtualHost> 

Any idea if this is at all possible?

Apache 504 and 502 timeouts over load balancer

Brief

I am using linode as my servers and just added a node for Load Balancing. Main part of my portal is that it downloads the heavy files from AWS to the server, creates the zip there and then that zip is downloaded onto my local machine.

Problem After moving it to the Load Balancer, I am facing 504 Bad Gateway and 502 Bad Gateway errors on heavy downloads (i.e. more than 100 MB) Here it is said that Load Balancers throw 504 when they wait for the response from Apache for too long.

Home work

I tried downloading on a server where there wasn't any LB configured, files were downloaded correctly. (on my local machine ), though it took a hell of a time but there wasn't any Bad Gateway error

I am using Apache with Ubunut 14.04 LTS and all the parameters for Apache like max_time_out etc etc are set to unlimited