Elgamal encryption algorithm

2023-01-08   ES  

Reference blog and document




griffin0.4.0 Installation

Installation steps
Installation dependencies
Declax griffin compression package
Create Griffin users in MySQL
Griffin dependency table creation
Hadoop and HIVE
livy configuration
elasticsearch configuration
GRIFFIN configuration file modification
application.properties configuration
Quartz.properties configuration
sparkproperties.json configuration
env_batch.json configuration
service/pom.xml file configuration (particularly important)
Use Maven command to compile and pack
copy jar package
The default port of the Spark Master Web UI
Start Hive source data service
Start Griffin
Started successful verification

Installation steps

Apache Griffin is the open source data quality solution of big data, which supports batch processing and flow mode. It provides a unified process that can measure your data quality from different angles, help you build trustworthy data assets, thereby improving your confidence in your business.
Installation dependencies

Apache Hadoop: Batch data source, storage indicator data
Apache Hive: Hive Metastore
Apache spark: Calculate batch, real -time indicators
Apache Livy: Provide the Restful API for the service to call Apache Spark
MySQL: Service metadata
Elasticsearch: Storage indicator data
Maven: Project Management Tool Software, which is used to pack the GRIFFIN project, followed by GRIFFIN to run in JAR. If it is a production library, put the JAR package in the platform to run the GRIFFIN in the platform because Maven runs because Maven is running. Many components will be installed, so the external network will be required.
Related links:
Hadoop Installation: https://blog.csdn.net/genus_yang/article/details/87917853
hive Installation: https://blog.csdn.net/genus_yang/article/details/879387966
spark Installation: https://blog.csdn.net/genus_yang/article/details/88018392
Livy Installation: https://blog.csdn.net/genus_yang/article/details/8802779999
MySQL Installation: https://blog.csdn.net/genus_yang/article/details/87939556
elasticsearch installation: https://blog.csdn.net/genus_yang/article/details/88051980
Maven download link:
virtual machine NAT connection external network link: https://blog.csdn.net/qq_40612124/article/details/7908427666
Declapping Griffin compressed package

    [[email protected] ~]$ unzip griffin-0.4.0-source-release.zip

Create Griffin users in MySQL

    [[email protected] ~]# mysql -u root -p123
    mysql> create user ‘griffin’ identified by ‘123’;
    mysql> grant all privileges on . to ‘griffin’@’%’ with grant option;
    mysql> grant all privileges on . to [email protected] identified by ‘123’;
    mysql> flush privileges;

griffin dependency table creation

Griffin uses the Quartz scheduler scheduling task. You need

    [[email protected] ~]# mysql -h master -u griffin -p123 -e “create database quartz “
    [[email protected] ~]# mysql -h master -u griffin -p123 quartz < /home/hadoop/griffin-0.4.0/service/src/main/resources/Init_quartz_mysql_innodb.sql

Hadoop and HIVE

# Create/home/spark_conf directory

    [[email protected] ~]$ hadoop fs -mkdir -p /home/spark_conf
# Upload the hive configuration file hive-size.xml
    [[email protected] ~]$ hadoop fs -put /home/hadoop/hive-3.1.1/conf/hive-site.xml /home/spark_conf/

livy configuration

update livy.conf configuration file under livy/conf

    [[email protected] ~]$ cd livy-0.5.0/conf/
    [[email protected] conf]$ vi livy.conf
    livy.server.host =
    livy.server.port = 8998
    livy.spark.master = yarn
    #livy.spark.deploy-mode = client
    livy.spark.deployMode = cluster
    livy.repl.enable-hive-context = true

Attachment of Yarn-CLUSTER and Yarn-Client mode: https://blog.csdn.net/zxr717110454/article/details/806365699

Start Livy:

    [[email protected]~] $ Livy-Server Start # Start Start Stop Status State

elasticsearch configuration

Start ES (may be a bit slow)
[[email protected] ~]$ ./elasticsearch-6.6.1/bin/elasticsearch
[[email protected] ~]$ ./elasticsearch-6.6.1/bin/elasticsearch
[[email protected] ~]$ ./elasticsearch-6.6.1/bin/elasticsearch
Create Griffin index in ES

    [[email protected] ~]$ curl -H “Content-Type: application/json” -XPUT http://master:9200/griffin -d ’

    “aliases”: {},
    “mappings”: {

    “accuracy”: {

    “properties”: {

    “name”: {

    “fields”: {

    “keyword”: {

    “ignore_above”: 256,
    “type”: “keyword”
    “type”: “text”
    “tmst”: {

    “type”: “date”
    “settings”: {

    “index”: {

    “number_of_replicas”: “2”,
    “number_of_shards”: “5”
The correct display result is:

If you do not add parameters -H “Content -Type: Application/JSON”, the error result is:
{“error”:“Content-Type header [application/x-www-form-urlencoded] is not supported”,“status”:406}
Griffin configuration file modification


Griffin directory includes four modules: Griffin-Doc, Measure, Service, and UI. Among them, Griffin-DOC is responsible for storing Griffin documents. Measure is responsible for interacting with Spark to perform statistical tasks. Service uses Spring Boot as a service. It is responsible for UI. The module provides the RESTFUL API required for interaction to save statistical tasks and display statistical results.
source code is introduced after the construction is completed, and the configuration file needs to be modified.

Enter the directory of the configuration file

    [[email protected] resources]$ cd /home/hadoop/griffin-0.4.0/service/src/main/resources

Due to the large number of parameters, I will shine thicken and modify it myself for the parameters to be modified.
Application.properties configuration

    [[email protected] resources]$ vi application.properties
#APACHE GRIFFIN application name
#Mysql database configuration information
#hive metastore configuration information
    #Hive cache time
#kafka schema registry, configure it on demand
    #Update job instance state at regular intervals
    #Expired time of job instance which is 7 days that is 604800000 milliseconds.Time unit only supports milliseconds
    #schedule predicate job every 5 minutes and repeat 12 times at most
    #interval time unit s:second m:minute h:hour d:day,only support these four units
    #external properties directory location
    #external BATCH or STREAMING env
    #login strategy (“default” or “ldap”)
#LDAP, the login strategy is configured when ldap
    [email protected]
    #hdfs default name
#Lasticsearch configuration
    #elasticsearch.user = user
    #elasticsearch.password = password
#Livy configuration
#yarn url configuration
    #griffin event listener

Quartz.properties configuration

    [[email protected] resources]$ vi quartz.properties
    #If you use postgresql as your database,set this property value to org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
    #If you use mysql as your database,set this property value to org.quartz.impl.jdbcjobstore.StdJDBCDelegate
    #If you use h2 as your database, it’s ok to set this property value to StdJDBCDelegate, PostgreSQLDelegate or others

sparkproperties.json configuration

    [[email protected] resources]$ vi sparkProperties.json

    “file”: “hdfs:///griffin/griffin-measure.jar”,
    “className”: “org.apache.griffin.measure.Application”,
    “name”: “griffin”,
    “queue”: “default”,
    “numExecutors”: 2,
    “executorCores”: 1,
    “driverMemory”: “1g”,
    “executorMemory”: “1g”,
    “conf”: {

    “spark.yarn.dist.files”: “hdfs:///home/spark_conf/hive-site.xml”
    “files”: [
No need to modify the default,
hdfs: ///griffin/griffin-measure.jar: The position uploaded by the measure jar package
hdfs: ///home/spark_conf/hive-site.xml: The location of the hive configuration file above

env_batch.json configuration

    [[email protected] resources]$ vi env/env_batch.json

    “spark”: {

    “log.level”: “WARN”
    “sinks”: [

    “type”: “CONSOLE”,
    “config”: {

    “max.log.lines”: 10

    “type”: “HDFS”,
    “config”: {

    “path”: “hdfs:///griffin/persist”,
    “max.persist.lines”: 10000,
    “max.lines.per.file”: 10000

    “type”: “ELASTICSEARCH”,
    “config”: {

    “method”: “post”,
    “api”: “http://master:9200/griffin/accuracy”,
    “connection.timeout”: “1m”,
    “retry”: 10
    “griffin.checkpoint”: []

service/pom.xml file configuration (particularly important)

Edit service/pom.xml file No. 113, remove mysql jdbc dependency annotation:

Otherwise, starting Griffin will report an error:

    nested exception is java.lang.Illeg
    alStateException: Cannot load driver class: com.mysql.jdbc.Driver

Use Maven command to compile and pack

After the

    [[email protected] ~]$ cd griffin-0.4.0/
    [[email protected] griffin-0.4.0]$ mvn -Dmaven.test.skip=true clean install

command is executed, you will see the service-0.4.0.jar and measure-0.0.jar in the target directory of the service and the Measure module, respectively.
copy jar package

Measure-0.4.0.jar renamed the name, the same as the name in SparkProperties.json

    [[email protected] ~]$ cd griffin-0.4.0/measure/target/
    [[email protected] target]$ mv measure-0.4.0.jar griffin-measure.jar

Create HDFS/GRIFFIN directory

    [[email protected] target]$ hadoop fs -mkdir /griffin/

Uploaded the renamed Griffin-MEASUR

    [[email protected] target]$ hadoop fs -put griffin-measure.jar /griffin/

Copy the service-0.0.jar to the main directory

    [[email protected] ~]$ cp /home/hadoop/griffin-0.4.0/service/target/service-0.4.0.jar .

The default port of the Spark Master Web UI

Because Griffin’s Sprint Root’s default startup port is 8080, conflict with Spark’s default port, you can change the SPARK port to avoid conflicts.

    [[email protected] ~]$ cd spark-2.4.0/sbin/
    [[email protected] sbin]$ vi start-master.sh
Positioning to line 61
    if [ “$SPARK_MASTER_WEBUI_PORT” = “” ]; then
Modify to other port number

Start Hive source data service

    [[email protected] ~]$ cd hive-3.1.1/
    [[email protected] hive-3.1.1]$ bin/hive –service metastore &

Otherwise, the Hive database will not be found when creating Measure, and the error is:

Start Griffin

Run the service-0.4.0.jar, start the GRIFFIN management background

    [[email protected] ~]$ nohup java -jar service-0.4.0.jar>service.out 2>&1 &

Started successful verification

Visit the UI interface, the following figure showed that the installation was successful!
Insert a picture description here
Author: yangjab
Source: CSDN
Original: https://blog.csdn.net/genus_yang/article/details/88053170
Copyright Statement: This article is an original article of bloggers. Please attach a blog post link for reprint!


Related Posts

Use spark, ANSJ segmentation to perform frequent statistics

Python 3.8 version, install and use the Pymouse library

[SpringCloud] Fenchley & Greenwich skeleton construction (1)

GO and Java how to handle the concurrent and intersection of the two arrays

Elgamal encryption algorithm

Random Posts

Ruby On Rails Oracle Configure Oracleenhanced, Ruby-OCI8 installation, ORA-12154: TNS

httpcomponents 4.5.2 Simple use

Retis distributed lock (optimistic lock) Nicolos based on LUA script

nodejs knowledge point

C ++ BFS Seeking the shortest path