Amazon EKS introduces Beta version of Amazon FSx for Lustre CSI Driver

The Amazon FSx for Lustre CSI driver has achieved beta status and is now sustained by Amazon Elastic Kubernetes Service (EKS). The CSI driver composes it elementary to design and utilise FSx for Lustre high functioning file systems with packages on EKS and autonomous Kubernetes clumps functioning on AWS. Amazon FSx for Lustre is an entirely counseled, high functioned file system enhanced for workloads. With FSx for Lustre, one can expeditiously and conveniently swirl up a high-performance file system associated with the S3 data archives, and access S3 objects as files. With the FSx for Lustre CSI driver, one can emphatically equip and mount an FSx for Lustre file system to packages, so that the containerised workloads can constitutionally access and operate data reserved in the file system or S3 data repository. One can utilise the CSI driver to mount and distribute the FSx file system over multiple cases from different nodes. Containerised operations that need high functioning storage can assist from using the FSx for the Lustre file system with the CSI driver. Such features comprise distributed machine learning workloads on the structures such as Tensor-flow and PyTorch, and media refining workloads. Administering high-performance file systems requires specially designed expertise and bureaucratic overhead, required to arrange the storage servers and tune convoluted performance criterion.

Amazon Athena provides assistance for User Defined Functions (UDF)

Amazon Athena is an interactive query service which makes simple to study data straight in Amazon S3 with the help of standard SQL. You do not need to handle any infrastructure as Amazon Athena is serverless. Athena is simple to utilize. Simply point to your data in Amazon S3, specify the schema, and begin querying using standard SQL to execute ad-hoc queries and you will get results within seconds with few clicks in AWS management console. There is no necessity for complex ETL jobs to make your data for analysis with Amazon Athena. This makes it simple to SQL expertise to faster explore large-scale datasets. Now Amazon Athena offers new feature i.e. user-defined functions (UDFs), a feature which allows users to write custom scalar functions and call them in SQL queries. While Amazon Athena presents built-in functions, UDFs allow users to execute custom processing like compressing and decompressing data, redacting sensitive data, or applying customized decryption. Using Athena Query Federation SDK users can write their UDFs in Java. When a UDF is used in a SQL query submitted to Athena, it is called and performed on AWS Lambda. UDFs can be utilized in both SELECT and FILTER clauses of a SQL query. Users can call several UDFs in the same query. This new Athena UDF feature is accessible in Preview mode in the us-east-1 (N. Virginia) region. You can follow these steps to start your Preview. To know more on User Define Functions (UDF), read documentation. You can get UDF example implementations here. And to know further how to write your functions with Athena Query Federation SDK, go through this link.

Amazon Redshift Spectrum introduces in 5 more AWS regions

Amazon Redshift Spectrum is a feature of Amazon Redshift that expands analytics to data saved in your Amazon S3 data lake, excluding the need of loading or transforming data. You can easily set up, automates majority of your administrative activities, and gives quick performance at any scale. Amazon Redshift Spectrum helps open data formats, like Parquet, ORC, JSON, and CSV. Further it also assists querying nested data with complex data types like struct, array, or map. You can query data over Redshift and Amazon S3 to get specific insights which are not feasible to get by querying independent datasets. You can execute analytic queries against petabytes of data saved locally in Redshift, and straight against exabytes of data saved in Amazon S3. Now this Amazon Redshift Spectrum feature is available in 5 more AWS Regions – EU (Paris, Stockholm), Middle East (Bahrain), Asia Pacific (Hong Kong), and the AWS GovCloud (US-West) Region. With these new 5 regions, Amazon Redshift Spectrum is accessible in 19 AWS Regions across the globe : US East (N. Virginia, Ohio), US West (Oregon, N. California), AWS GovCloud (US West), Canada (Central), South America (Sao Paulo), Middle East (Bahrain), EU (Frankfurt, Ireland, London, Paris, Stockholm), and Asia Pacific (Hongkong, Mumbai, Seoul, Singapore, Sydney, Tokyo). To get further information on Amazon Redshift Spectrum, read documentation.

Amazon GuardDuty helps Exporting Findings to Amazon S3 Bucket

Amazon GuardDuty is a threat detection service which non-stop observes for harmful and uncertified action to secure your AWS accounts and workloads. Amazon GuardDuty examine billions of events over your AWS accounts from AWS CloudTrail (AWS user and API activity in your accounts), Amazon VPC Flow Logs (network traffic data), and DNS Logs (name query patterns). The Amazon GuardDuty service is based on machine learning, abnormality detection, and integrated threat intelligence to spot and prioritize potential threats. GuardDuty informs you of the status of your AWS environment by producing security findings that you can view in the GuardDuty console or through Amazon CloudWatch events. Now Amazon GuardDuty users can export findings to Amazon S3 with the help of GuardDuty management console and API. Aggregating findings from across regions is clarified with findings export. After configured from the GuardDuty master account, users can export findings from every linked member accounts and all AWS regions to one user held S3 bucket. The utilized S3 bucket can be in the same account in which GuardDuty is activated, or in any other AWS account. Once Findings export is configured in each Region, Amazon GuardDuty findings are automatically exported from GuardDuty to the configured Amazon S3 bucket. To know further about Findings export, refer GuardDuty User Guide and Amazon GuardDuty Findings. To get the complete list of AWS Region where Amazon GuardDuty is accessible, visit AWS Regions. You can start your 30-day Amazon GuardDuty Free Trial in the AWS Management console with just few clicks.

Now Amazon Transcribe’s Speech-to-text feature is available in 8 further languages

Amazon Transcribe is an automatic speech recognition (ASR) service that based on advanced machine learning technologies which can use to convert audio files into text format and to build applications that incorporate the content of audio files. Amazon Transcribe can be utilized for plenty applications. For example, you can transcribe the audio track from a video recording to generate subtitles for the video. Amazon Transcribe can transcribe audio files saved in common formats, like WAV and MP3, with time stamps for each word so that you can simply find the audio in the original source by looking for the text. Amazon Transcribe is constantly learning and enhancing to boost the progress of language. You can examine audio files stored in Amazon S3 with the help of Amazon Transcribe API. Besides, you can send a live audio stream to Amazon Transcribe and get a stream of transcripts in real time. Now Amazon Transcribe is available in further 8 languages - Irish English, Scottish English, Welsh English, Dutch, Farsi, Indonesian, Portuguese, and Tamil languages. With these new supported languages, Amazon Transcribe is now totally available in Modern Standard Arabic, Chinese Mandarin-Mainland, Dutch, Australian English, British English, Indian English, Irish English, Scottish English, US English, Welsh English, French, Canadian French, Farsi, German, Indian Hindi, Indonesian, Italian, Korean, Portuguese, Brazilian Portuguese, Russian, Spanish, US Spanish, and Tamil languages. To get the further information, refer Amazon Transcribe or Documentation.

Amazon Athena is Accessible in Asia Pacific (Hong Kong) Region

Amazon Athena is a simple interactive query service which examines data straight in Amazon S3 with the help of standard SQL. Athena is integrated with AWS Glue Data Catalog, enabling you to design a unified metadata repository over different services, crawl data sources to locate schemas and occupy your Catalog with latest and altered table and partition definitions, and maintain schema versioning. There is no need to set up or handle any infrastructure as Amazon Athena is serverless. And you only charge for the queries you execute. You get the faster results as Athena scales automatically—running queries in concurrently with big datasets and complex queries. You can point Athena at your data stored in Amazon S3 with few clicks in AWS Management Console, and start using standard SQL to execute ad-hoc queries and get results within seconds. Now Amazon Athena is accessible in Asia Pacific (Hong Kong) AWS Region. With this region availability, Amazon Athena is accessible in US East (Northern Virginia, & Ohio), US West (Oregon), AWS GovCloud, Europe (Ireland, Frankfurt, London, & Stockholm), and Asia Pacific (Singapore, Tokyo, Sydney, Seoul, Mumbai, & Hong Kong) AWS Regions.

AWS Snowball Edge is accessible in Asia Pacific (Hong Kong)

AWS Snowball Edge is a service available in two options for a data migrations and edge computing device. AWS Snowball Storage Optimised offers both block storage and Amazon S3-compatible object storage, and 24 vCPUs. Besides, it it right choice for local storage and large scale-data transfer. Also Snowball Edge Compute Optimized gives 52 vCPUs, block and object storage, and an optional GPU for use cases like advanced machine learning and full motion video analysis in disconnected environments. Snowball Edge assists particular Amazon EC2 instance types and also AWS Lambda functions, so customers may develop and test in AWS then deploy applications on devices in remote locations to collect, pre-process, and return the data. Now AWS Snowball Edge is accessible in the Asia Pacific (Hong Kong) AWS region. Snowball Edge supports to solve the obstacles migrating huge data sets to the cloud when you do not have adequate network bandwidth for transfers. For edge computing use cases, Snowball Edge allows you to execute Amazon EC2 instances and AWS Lambda functions on AWS IOT Greengrass. To get the full list of AWS Region where AWS Snowball Edge is available, refer AWS Region table. To get more information, read documentation or visit Snowball Edge web page, AWS Console.

AWS DataSync is accessible in the Middle East (Bahrain) Region

AWS DataSync is completely organized service. AWS DataSync is a data transfer service which clarifies, automates, and boosts moving and replicating data between on-premises storage systems and Amazon S3 or Amazon EFS, AWS storage services across the internet or AWS Direct Connect. AWS DataSync excludes the requirement to alter applications, develop scripts, or manage infrastructure. AWS DataSync works with on-premises software agent to link to your current storage or file systems with the help of Network File System (NFS) and Server Message Block (SMB) protocols, so you do not have to write scripts or alter your applications to work with AWS APIs. AWS DataSync can use for relocation of active data, distribution of data for analysis and processing in the cloud, or ongoing replication to AWS for business continuity. Now AWS DataSync to transfer data into and out of Amazon S3 buckets is available in the AWS Middle East (Bahrain) Region. With this new region, currently AWS DataSync is obtainable in the following AWS Regions : US East (Northern Virginia and Ohio), US West (Northern California and Oregon), AWS GovCloud (US-West), Europe (Ireland and Frankfurt), Middle East (Bahrain), and Asia Pacific (Singapore, Tokyo, Sydney, and Seoul).

Now Amazon Athena helps querying data in Amazon S3 Requester Pays buckets

Amazon Athena is an interactive query service which makes it simple to examine data straight in Amazon Simple Storage Service (Amazon S3) with the help of standard SQL. You can point Athena at your data stored in Amazon S3 using few clicks in the AWS Management Console and start utilizing standard SQL to execute ad-hoc queries and obtain results within seconds. There is no infrastructure to set up or manage as Athena is serverless, so you just charge for the queries you execute. Athena scales automatically—running queries simultaneously —so results are fast, even with large datasets and complex queries. Now Amazon Athena helps querying data in Amazon S3 Requester Pays buckets. Using this new launched feature, Athena workgroup administrators can configure workgroup settings to permit members to reference S3 Requester Pays buckets in queries. Once configured, the requester, rather than the bucket owner, pays for the Amazon S3 request and data transfer charges related to the query. Refer Create a Workgroup from Amazon Athena User Guide to know how to configure this setting for your workgroup. And to read further on Requester Pays buckets, click Requester Pays Buckets in the Amazon Simple Storage Service Developer Guide.

Amazon Kinesis Data Firehose is obtainable in the Asia Pacific (Hong Kong) AWS Region

Amazon Kinesis Data Firehose makes it simple to precisely load streaming data into data lakes, data stores and analytics tools. This helps to capture, transform, and load streaming data into Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, and Splunk, allowing near real-time analytics with current business intelligence tools and dashboards you’re already using today. It is a completely organized service which automatically scales to match the throughput of your data and needs no continuous administration. Further it can batch, compress, transform, and encrypt the data before loading it, reducing the amount of storage used at the destination and maximizing security. Now this Amazon Kinesis Data Firehose is accessible in the Asia Pacific (Hong Kong). You don't require to write applications or manage resources with Amazon Kinesis Data Firehose. Additionally, you can also configure Amazon Kinesis Data Firehose to transform your data before delivering it. You can create a delivery stream in the Amazon Kinesis Console. To get more information about Amazon Kinesis Data Firehose, refer documentation. To get the complete list of Amazon Kinesis Data Firehose availability, refer to the AWS Region Table.

Amazon SageMaker Batch Transform Allows Forecast Results

Amazon SageMaker Batch Transform allows you to execute forecasts on datasets saved in Amazon S3. It is perfect for situations where you are functioning with sizeable batches of data and do not require sub-second latency. Now you can now configure your Batch Transform Jobs to remove particular data attributes from forecast requests, and to connect some or all of the input data attributes with forecast results. As an outcome, you no further require extra pre-processing or post-processing when executing batch forecasts on data which is in CSV or JSON format. Take an example, suppose a dataset which contains three attributes: ID, age, and height. The ID attribute is indiscriminately created or sequential number which brings no signal for the ML problem and was not utilized when teaching the ML model. You can configure your Batch Transform jobs to remove the ID attribute from all record, and send only the age and height attributes in the forecast requests sent to the model and also to link the ID attribute with the forecast results in the last S3 output of the job. Keeping record-level attributes in this manner can be helpful for examining the forecast outcomes. This new feature is accessible in each region where Amazon SageMaker is accessible. To get the complete list of AWS Region where Amazon SageMaker is available, refer AWS Region Table. To get further detail about this feature, read Amazon SageMaker

Now Amazon Athena Is Accessible In GovCloud (US-East)

Amazon Athena is an interactive query service which is easy to use and simple to study data in Amazon S3 with the help of standard SQL. Athena is serverless, in order that there is no infrastructure to handle, and you charge exclusively for the queries which you execute. Just indicate to your data in Amazon S3, specify the schema, and begin querying with the help of standard SQL. Many outcomes are offered in few seconds. There is no necessity for complex ETL (Extract, Transform, Load) jobs to make your data for analysis with Amazon Athena. This helps for everybody with SQL expertise to rapidly study large-scale datasets. Athena is merged with AWS Glue Data Catalog, permitting you to build a unified metadata repository over different services, crawl data sources to locate schemas and occupy your Catalog with new and modified table and partition definitions, and retain schema versioning. Besides, you can utilized AWS Glue’s completely organized ETL potentials to change data or convert it into columnar formats to optimize cost and better performance. Now Amazon Athena is accessible in the AWS GovCloud (US-East) region and US East (Northern Virginia, & Ohio), US West (Oregon), AWS GovCloud, Europe (Ireland, Frankfurt, & London), and Asia Pacific (Singapore, Tokyo, Sydney, Seoul, & Mumbai) regions.

Now Amazon RDS for SQL Server Offers Assistance To SQL Server Audit

Amazon RDS for SQL Server is easy to set up, operate, and scale SQL Server deployments in the cloud. You can deploy multiple editions of SQL Server (2008 R2, 2012, 2014, 2016, and 2017) containing Express, Web, Standard and Enterprise with Amazon RDS, in minutes with cost-efficient and re-sizable compute capacity. Amazon RDS lets you concentrate on application development by handling time-consuming database administration activities covering provisioning, backups, software patching, monitoring, and hardware scaling. Now, Amazon RDS for SQL Server offers assistance to SQL Server Audit which allows you build server audits, that includes server audit specifications for server level events, and database audit specifications for database level events. You can immediate access the audits on the database instance and automatically send the audit log files straight to Amazon S3. To know further on how to setup and configure SQL Server Audit, refer Amazon RDS User Guide and Amazon RDS Pricing for pricing and regional availability.

Cheapest Solution On FTP Over AWS Cloud

Businesses are always helped by the evolving software tools in respect to the increasing demands of the market. Apart from secure working mechanism, it would be the cherry on the top if the cost implications for such softwares are cheap enough. In this post we will go through one of such files upload/download tool/service provided by AWS which is secure and cheapest as compare to other cloud technologies.

FTP (File Transfer Protocol) is a fast and handy way to transfer small/large files over the Internet. At some point, we may have configured an FTP server backed up by block storage, NAS, or a SAN. However, involving this kind of backend storage options requires infrastructure support and can also cost a fair amount of time and money.

Why S3 FTP?
Amazon S3 service is reliable and have user friendly interface. Amazon S3 features during the last edition of re:Invent.

• Amazon S3 offers an infrastructure that’s “designed for durability of 99.999999999% of objects.”
• Amazon S3 is designed to provide “99.99% availability of objects over a year.”
• You pay for exactly what you need with no minimum commitments or up-front fees.
• With Amazon S3, we can store unlimited data you can store or when you can access it.

S3 FTP : Implementation

1. Using S3 : Object Storage As Filesystem :
Create a S3 bucket that will be used as filesystem, which can be done by AWS console or API.

2. IAM Policy And Role :
Create an IAM Policy and Role to control access into the previously created S3 bucket which also can be done by AWS console or API.

3. FTP Server :
Launch a EC2 instance that will be used for hosting FTP service.

4. Setting Up S3FS On FTP Server :
We will configure S3FS on the FTP server in order to mount the S3 bucket as file system. Here we can follow the below steps to configure the same.

Step-1 :- If you are using a new centos or ubuntu instance. Update the system

• For CentOS or Red Hat
    # yum update all
• For Ubuntu
    # apt-get update

Step-2 :- Install the dependencies.

• For CentOS or Red Hat
    # sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
• For Ubuntu
     # sudo apt-get install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config

Step-3 :- Clone s3fs source code from git.

• git clone https://github.com/s3fs-fuse/s3fs-fuse.git

Step-4 :- Now navigate to source code  directory, and compile and install the code with the following commands:

• cd s3fs-fuse
• ./autogen.sh
• ./configure --prefix=/usr --with-openssl
• make
• sudo make install

Step-5 :- Use below command to check where s3fs command is placed in O.S. It will also tell you the installation is ok.

• which s3fs

5. Configure FTP User Account And Home Directory :
Create a ftptest user account which we will use to authenticate against our FTP service:

• sudo adduser ftptest 
• sudo passwd ftptest

Now create the directory structure for the ftptest user account which we will later configure within our FTP service, and for which will be mounted to using the s3fs:

• sudo mkdir /home/ftptest/ftp
• sudo chown nfsnobody:nfsnobody /home/ftptest/ftp
• sudo chmod a-w /home/ftptest/ftp
• sudo mkdir /home/ftptest/ftp/files
• sudo chown ftptest:ftptest /home/ftptest/ftp/files

6. Install And Configure vsftpd Over The Server :
Now install and configure our FTP service with the vsftpd package:

• sudo yum -y install vsftpd

7. Startup S3FS and Mount Directory :
We will configure S3FS to mount the S3 bucket using below commands:

1. Gather IAM credentials for required S3 bucket access / full S3 access.
2. Create a  file in /etc with the name passwd-s3fs and paste the access key and secret key in the below format.
• vi /etc/passwd-s3fs
       # Your_accesskey:Your_secretkey
3. Change the permission of file.
• sudo chmod 640 /etc/passwd-s3fs
4. Mount the bucket on the directory created in Step 5
• sudo s3fs your_bucketname -o use_cache=/tmp -o allow_other -o uid=1001 -o mp_umask=002 -o multireq_max=5 /home/ftptest/ftp/files
• vi /etc/rc.local
• sudo s3fs your_bucketname -o use_cache=/tmp -o allow_other -o uid=1001 -o mp_umask=002 -o multireq_max=5 /home/ftptest/ftp/files

Atlast we can test if the S3 bucket is mounted successfully on the desired folder over the server or not.

•  df -h

Also we can connect and test all the setup through filezilla or any other tool for uploading/downloading the files to AWS S3 using S3FS.
 
With this blog description we have witnessed – how we can leverage the S3FS together with both S3 and FTP to build a file transfer solution! If you have any queries then write us at support@cloud.in

How ETL Operates With Amazon Glue

AWS Glue is a fully managed ETL (extract, transform, and load) service that can categorize your data, clean that data, enrich it, and move it between various data stores. AWS Glue consists of a central data repository which is known as the AWS Glue Data Catalog, an ETL engine which automatically generates Python code, and a scheduler which handles the dependency resolution, job monitoring and retries. AWS Glue is server-less, so there's no infrastructure to manage.

It signifies that you just have to concentrate on building your jobs and scripting your business logic, rather than building servers, installing tools and ensuring the focus on when jobs need to run.

Event-Driven Or Scheduling

You can either schedule your jobs to run at predefined intervals, or you can have them run based on triggers on S3 buckets. For example you can set up a Lambda function to trigger your job whenever a new file is dropped in specific bucket.

What It Does?
1. It collects information about your data sources. This includes where the data is stored, and the underlying schema of that data.

2. It builds transformations between data sources. AWS Glue uses crawlers to inspect your variable schemas, and auto-generates the necessary code to transform from source to destination.

3. It manages Jobs to move the data, allowing for powerful scheduling and retry possibilities.

4. It seamlessly integrates with other AWS Services, including S3 and Amazon Redshift Spectrum.

How It works?

Setup The Crawler

With having data in hand, the next step is to point AWS Glue Crawler to data. The crawler inspects the data and generate a schema describing what it finds. While AWS Glues supports various custom classifiers for complicated data sets.

Create A Job

With the schema in place, we can create a Job. We don't need any fancy scheduling here, just need it to execute.

AWS Glue offers a GUI to define your input/output mappings, or you can just edit the script directly. For this simple example, I removed some of the output fields (so we're effectively reducing the number of columns in our output data set)

Upon successful completion of our job, we now have a (transformed) data set in our S3 storage!

You can write your own ETL scripts using Python or Scala.

S3 Data Into AWS RedShift

AWS Redshift is a powerful Data Warehouse solution, and perfect for our needs. Utilising the "COPY" command, we can easily copy our data into AWS Redshift which is transferred to glue from s3.

This shows us how easy, fast and scalable it is to crawl, merge and write data for ETL operations using Glue, a very good service provided by Amazon Web Services.