Google Cloud in PHP

At this post I will give some examples of how to use Google APIs Client Library for PHP in order for interact with Google Cloud. Google BigQuery API client library for PHP First you need to download Google BigQuery API client library for PHP (download Google BigQuery API client library here). Open it and use […]

Bash script to upload from Google Cloud Storage to Google BigQuery

The following Bash script is using the Google tool to upload logs files (statistics) from Google Cloud to Google BigQuery. The script uploading several logs files to one table and has the ability to manipulate the data and insert it to new table at BigQuery. For uploading the logs to the Google Cloud please […]

Bash script to upload logs to Google Cloud Storage

The following Bash script is using the Google Gsutil tool to upload hourly rotated server logs files (statistics) to Google Cloud. For a Bash script to upload logs files from Google Cloud to Google Bigquery please read Bash script to upload from Google Cloud storage to Google Bigquery. Because the logs are rotating every hour, […]

Google BigQuery API Client Example in PHP

Here is an example of how to use Google APIs Client Library for PHP in order for interact with Google BigQuery. First you need to download Google BigQuery API client library for PHP (download from here). Open it and use only the Google directory. Before coding, please validate that you have a Google BigQuery account […]

java.lang.NoClassDefFoundError when loading a class located in another project at Eclipse

It is useful to use classes that resides in another project in the Eclipse Workspace. In order to avoid java.lang.NoClassDefFoundError error, perform the following steps: Add the project that has the wanted classes to the Java Build Path Properties-> java build path -> projects ->add your project that contain wanted classes Check the added project […]

How to Build a RESTful Web Service with JAX-WS and Jersey on Eclipse

In order to build a RESTful API with Jersey on Eclipse, please use the following steps: Create new Eclipse Dynamic Web Project Open Eclipse, then File -> New -> Dynamic Web Project. Name it “TestProject”. Generating web.xml file Generate a web.xml file IN WebContent/WEB_INF by: Right click on the created project -> Java EE Tools […]

Deleting batch of rows from MS-SQL Server table

Deleting large number of rows from a (SQL Server) table can cause replication delay, create a very large transaction and a big performance impact, as well as escalating locks so that the table will be unavailable. If it is possible use truncate: TRUNCATE TABLE MyTable; Truncate will run much faster although it has no facility […]

Hibernate: How to insert OneToMany children by cascade

I needed a way to insert an entity with a new joined (children) entity together. One way to do it is by separate the actions (e.g. to save the joined entity first and then to save the main entity). However, Hibernate support inserting a new entity with new joined entity. I would like to present […]

Solr spellcheck compound from several fields

The Solr’s SpellCheck component is designed to provide inline spell checking of queries (i.e. query suggestions or “Did You Mean”) in case it thinks the input query might have been misspelled. The words can be loaded from text files, a field in Solr, or even from several fields (more than one field). In order for […]

How to create a copy of existing Solr index

If you have an already existing Solr index and you want to try new settings, it is best to work and test on a copy Solr’s index first. Here are the simple steps to make a copy of your Solr’s index: Suppose we have already an index named cast and the copy will be named […]

Drupal Sitemap does not show URL aliases

The Sitemap module The Drupal Sitemap module generates a sitemap page and gives visitors a site overview. All categories and terms (i.e. vocabulary) are expanded optionally with node counts and RSS feeds. The taxonomy URL paths on the sitemap page are not in the URL paths’ alias format In Drupal 6, I encountered a problem: […]

How to log all MySQL queries in Drupal

In order to benchmark a Drupal site performance you need to see all the database queries related to your Drupal site. In case you don’t have access to the my.cnf file If you don’t have access to the my.cnf file, you can log the queries from the Drupal code itself: cp includes/ includes/ vim includes/ […]

Insert into multiple MySQL tables with one-to-one relationship

I have been asked if there is a way to insert data into two MyISAM tables that connected by a one-to-one relationship in one MySQL query. For example: mysql> CREATE TABLE `user` ( `id` int(10) unsigned NOT NULL auto_increment, `name` varchar(255) NOT NULL default ”, PRIMARY KEY (`id`) ) ENGINE=MyISAM AUTO_INCREMENT=5 DEFAULT CHARSET=latin1   mysql> […]

Connection Pool: MySQL Communications link failure

The Problem And The Solution While using a MySQL connection pool in Java, I received a MySQL Communications link failure Exception (see below). In order to solve communication link failure exception: I have removed JDBC property autoReconnect=true and put only the JDBC property autoReconnectForPools=true I have added the connection properties: testOnBorrow testWhileIdle timeBetweenEvictionRunsMillis minEvictableIdleTimeMillis See […]

Connection Pool: MySQL connection closed exception

The Problem While using a MySQL connection pool in Java, I received a MySQL Connection Closed Exception (see below). The problem was that the JDBC driver was not compatible to the MySQL server version. I have used version 5.05a for both JDBC and MySQL and it solved the problem The problem is that the connection […]

How To Switch Java in Centos

* If you need to install a new version of java, download the version from java.sun * run /usr/sbin/alternatives to change the default java. If you will run the next command you will see which java versions are available: # /usr/sbin/alternatives –config java There is 1 programs which provide ‘java’. Selection Command ———————————————– *+ 1 […]

Adding DbUnit to your project

About DbUnit DbUnit is a JUnit extension (also usable with Ant) targeted at database-driven projects that, among other things, puts your database into a known state between test runs. This is an excellent way to avoid the myriad of problems that can occur when one test case corrupts the database and causes subsequent tests to […]

How to get directories via FTP

How to GET directories via FTP: You can get a directory files (recursive) by using wget command with flag -m for –mirror wget -m See using wget to recursively download whole ftp directories

How to setup log rotation on Flume

Flume can be used to rotate logs file into a regular file system. Examples: In BE mode (Best Effort) exec config ‘tail("/tmp/error.log")’ ‘agentBESink("",35888)’ exec config ‘collectorSource( 35888 )’ ‘collectorSink("/tmp/nlog/%Y/%m/%d/errorlog", "mylog_%{host}_%H_",60000 )’ in E2E (End to End) exec config ‘tail("/tmp/error.log")’ ‘agentE2ESink("",35888)’ exec config ‘collectorSource( 35888 )’ ‘collector(600000) {collectorSink("/tmp/nloge/%Y/%m/%d/errorlog", "mylog_%{host}_%H_",600000 )}’ Explanations: collectorSink […]

How to use the Flume E2E mode

E2E provides the best reliability mode: You can tune the reliability level of the agents, simply by specifying a different kind of agent sink. There are three levels available. One of them is the E2E (End To End) mode, which relies on an acknowledgement, and will retry if no acknowledgement is received. E2E limitations: E2E […]

How to set up a MySQL connection pool in Java

MySQL connection pool A MySQL connection pool is a pool of connections to MySQL database. Opening and maintaining a database connection for each process (or thread), is time costly (connection creation time) and wastes resources (connections). Connection pool increase the performance of (Java) applications that needs to connect to the database by reusing the connections. […]

How to install ANT on LINUX

Apache Ant is a Java library and command-line tool that help building software. How to install: Create downloads directory if you dont have one Download tar version from Extract Rename the new directory as ant Insert the path into ANT_HOME Update the global PATH variable to include ANT_HOME Run the Ant script fetch.xml to […]

How to rename a database in MySQL

In MySQL there is no support for database renaming. In order to rename a MySQL database you can do one of the following: 1. Create new database and rename all tables in the old database to be in the new database: CREATE database new_db_name; RENAME TABLE db_name.table1 TO new_db_name, db_name.table2 TO new_db_name; DROP database db_name; […]

Don’t Alter Table. Do Copy and Rename

In some cases a MySQL MyISAM table structure needs to be alter. This includes adding, removing and changing table columns (or indexes) and even re-ordering the MySQL table. In these cases, for performance and safety reasons, it is wise to avoid altering the current working MySQL table and adopt the Copy and Rename approach. The […]

Some powerful strengths of Regular Expression

Regular expressions provide a powerful, concise and flexible means for matching strings of text, such as particular characters, words, or patterns of characters. I will list here some powerful strengths: Word Boundaries The meta character \b is an anchor like the caret and the dollar sign. It matches at a position that is called a […]

How to select random rows in MySQL

The easiest way to generate random rows in MySQL is to use the ORDER BY RAND() clause. SELECT col1 FROM tbl ORDER BY RAND() LIMIT 10; This can work fine for small tables. However, for big table, it will have a serious performance problem as in order to generate the list of random rows, MySQL […]

How to dynamically create an object in Java from a class name given as string format

To create an object dynamically from the class name, you need to use a reflection. If the fully-qualified name of a class is available, it is possible to get the corresponding Class using the static method Class.forName(). However, This cannot be used for primitive types. Having the Class type, you need to use the newInstance() […]

How to determine the leading prefix part of a MySQL index

In MySQL, an index can be limited to use only a leading prefix part of a column values. Prefixes can be specified for CHAR, VARCHAR, BINARY, and VARBINARY columns, while for BLOB and TEXT columns the prefix must be given. For example, the statement shown here creates an index using the first 20 characters of […]

How to embed Flume node into your Java application

You may need to embed Flume node into your Java Program in a case you would like your application to control the incoming logs (you can then create new Sinks or Sources in the application). Embedding Flume into Java Application is possible and very easy to do. Instructions: 1. Install flume node 2. Configure your […]

MySQL select count

COUNT function counts the number of rows retrieved by a SELECT statement. The return value is of type BIGINT. The COUNT(expr) function count the number of non-NULL values of expr in the rows retrieved by a SELECT statement. SELECT COUNT(expr) FROM tbl; In contrast, COUNT(*) is different in that it returns a count of the […]

How to run the Flume’s HelloWorld example

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. For more information see extending_via_sink_source_decorator_plugins The flume’s Hello World example is an example of an experimental plug-in mechanism that allows to add new custom sources, sinks, and decorators to the Flume system. 1. The source code […]

How to install Flume on Linux

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. For more information see Flume Installation First you need to update your Yum repository: cd /etc/yum.repos.d/ sudo wget In order to install Flume node: sudo yum install flume-node -y In order to install Flume master: […]