发现问题

据说mariadb和mysql 90% 相似度,本地使用docker 起了个容器 挂载数据到本地的时候居然报错了

[ERROR] InnoDB: The Auto-extending innodb_system data file './ibdata1' is of a different size 0 pages than specified in the .cnf file: initial 768 pages, max 0 (relevant if non-zero) pages!

查询问题

和 mysql 一样的配置为什么会报错的,一脸懵逼??? ./ibdata1
查询了各大资源后发现可能是 Docker for windows 的共享文件系统或MariaDB本身中的bug!!!

解决问题

最后经过苦苦测试 终于找到了一个可行的解决方案
下面解释一下关键配置

  • ibdata1是InnoDB的共有表空间,默认情况下会把表空间存放在一个文件ibdata1中
  • innodb_flush_method 则确定日志及数据文件如何write、flush
  • innodb_flush_method可以取如下值:fdatasync, O_DSYNC, O_DIRECT 实验选项littlesync和nosync
  • innodb_use_native_aio 该选项仅适用于Linux系统,默认情况下处于启用状态。在其他类Unix系统上,InnoDB仅使用同步I/O. - 从历史上看,InnoDB仅在Windows系统上使用异步I/O. 在Linux上使用异步I/O子系统需要libaio库
  • Binary Log 简称 Binlog 即 二进制日志文件,这个文件记录了mysql所有的 DML 操作。
  • 通过 Binlog 日志我们可以做数据恢复,做主主复制和主从复制等等。

下面是的我的 docker-compose.yml 和 配置

dockerFile
FROM mariadb:latest

LABEL maintainer="Mahmoud Zalt <mahmoud@zalt.me>"

COPY my.cnf /etc/mysql/conf.d/my.cnf



CMD ["mysqld"]

EXPOSE 3306
docker-compose.yml
    mariadb:
      build: ./mariadb
      volumes:
        - ./mariadb/data:/var/lib/mysql
 
      ports:
        - "3306:3306"
      environment:
        - MYSQL_DATABASE=default
        - MYSQL_USER=default
        - MYSQL_PASSWORD=default
        - MYSQL_ROOT_PASSWORD=root
      networks:
        - backend
      command:
        mysqld --innodb-flush-method=littlesync --innodb-use-native-aio=OFF --log_bin=ON
mariadb 配置
[mysqld]
innodb_flush_method=littlesync
innodb_use_native_aio=OFF
log_bin=ON

安装包 (win10并开启虚拟化)

https://download.docker.com/win/stable/Docker%20for%20Windows%20Installer.exe

git 仓库

git@github.com:weiyixi/docker_study.git
目录说明
  1. nginx 站点配置在 ./docker-lnmp/nginx/site (修改后重启nginx容器生效)
  2. php-fpm 容器中/var/www/目录挂载本地docker-lnmp的上层目录(.env中 APPLICATION=../)
  3. 需要加扩展在对应得dockerfile 中修改

1.修改配置

env-example  复制为 .env
//window需要修改
DOCKER_SYNC_STRATEGY=unison
//php版本修改默认7.2
PHP_VERSION=72

2.基本操作

首先到docker-lnmp目录下

1.构建  docker-compose build  nginx php-fpm  redis  mysql 
2.开启  docker-compose up -d  nginx  redis  mysql 
3.关闭  docker-compose down
4.查看运行中容器  docker  ps
5.进入容器中   docker exec -it 对应的CONTAINER_ID  bash

3.扩展

1.已经定义了一批扩展可在 .env中开启 true开启 false 关闭
2.未定义的需要修改dockerfile安装
3.修改后需要重新build 开启才会生效

适用于对docker有一定了解的朋友,小白请先仔细阅读官方文档!
作为应用程序,我们通常需要依赖于多种外部服务,比如数据库、缓存服务等等。
Docker-compose就是在Docker容器的基础上,提供了统一的容器编排语言,可以让你更轻松的利用Docker构建你的应用环境。

docker-compose.yml文件语法语法

image 使用的镜像
build 指定Dockerfile构建
command 启动执行命令
links 链接其他容器    
ports 端口映射
expose 暴露端口
volumes 挂载路径
volumes_from 从容器挂载
environment 环境变量
  1. 编写dockerfile(自定义镜像)
  2. 编写docker-compose.yml(可以不依赖Dockerfile 使用官方镜像)

常用命令

构建容器 docker-compose up -d
查看容器 docker-compose ps
进入容器 docker exec -it 容器名 /bin/bash
关闭所有 docker-compose  down

php-fpm Dockerfile 示例 可以自定义扩展

FROM php:7.1-fpm-alpine

# apk
RUN apk --update add \
        autoconf \
        build-base \
        linux-headers \
        libaio-dev \
        zlib-dev \
        curl \
        git \
        subversion \
        freetype-dev \
        libjpeg-turbo-dev \
        libmcrypt-dev \
        libpng-dev \
        libtool \
        libbz2 \
        bzip2 \
        bzip2-dev \
        libstdc++ \
        libxslt-dev \
        openldap-dev \
        imagemagick-dev \
        make \
        unzip \
        wget \
        libmemcached-dev

# PHP Core Extensions
RUN docker-php-ext-install \
        bcmath \
        pdo_mysql \
        mysqli \
        pcntl \
    && docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ \
    && docker-php-ext-install gd

# PECL Extensions
RUN pecl install redis-4.2.0 \
    && pecl install mongodb-1.5.3 \
    && pecl install memcached-3.1.3 \
    && pecl install swoole-4.3.1 \
    && docker-php-ext-enable redis mongodb memcached swoole

# Delete
RUN apk del build-base \
        linux-headers \
        libaio-dev \
    && rm -rf /var/cache/apk/* \
    && mkdir /data \
    && chmod -R 777 /data

VOLUME /var/www
WORKDIR /var/www

EXPOSE 9000
CMD ["php-fpm"]

docker-compose.yml示例

version: "3.0"
services:
  nginx:
    image: nginx:alpine
    ports: 
      - 80:80
      - 443:443
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/conf.d:/etc/nginx/conf.d
      - ../:/var/www

  php-fpm:
    build: ./php-fpm
    ports:
        - 9000:9000
    volumes:
      - ./php-fpm/php.ini:/usr/local/etc/php/php.ini
      - ./php-fpm/www.conf:/usr/local/etc/php-fpm.d/www.conf
      - ../:/var/www

  mysql:
    image: mysql:5.7
    ports:
      - 3306:3306
    volumes:
      - ./mysql/data:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=123456

  redis:
    image: redis:5.0-alpine
    ports:
      - 6379:6379
    volumes:
      - ./redis/data:/data
      #- ./redis/redis.conf:/usr/local/etc/redis/redis.conf

  mongo:
    image: mongo
    ports: 
      - 27017:27017
    volumes:
      - ./mongo/data:/data/db
    restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: 123456

<script>
    /**
     * 数组中此元素出现的次数
     * @param search
     * @param array
     * @returns {number}
     */
    function in_array_count(search, array) {
        var num = 0;
        for (var i in array) {
            if (array[i] == search) {
                num++;
            }
        }
        return num;
    }

    /**
     * 数组中此元素是否出现
     * @param search
     * @param array
     * @returns {boolean}
     */
    function in_array(search, array) {
        for (var i in array) {
            if (array[i] == search) {
                return true;
            }
        }
        return false;
    }

    /**
     * 获取全部组合
     * @returns {Array}
     */
    function getProperty() {
        var property_name = ['颜色','颜色','尺寸','尺寸','尺寸','内存','内存','厚度','厚度'];         //属性名数组
        var property_value= ['红色','黑色','3.5','5.5','6.5','64G','128G','10mm','20mm'];
        var property_name_unique = [];  //去重的属性名
        var property_value_unique = []; //每个属性名对应的值  二维数组
        var property_name_val_count = [];//每个属性名对应的值个数
        var var_total_group = 1;         //组合的数量

        //所有的属性值
        for(value_key in property_value){
            if (!in_array(property_name[value_key], property_name_unique)) {
                property_name_unique.push(property_name[value_key]);//去重后的属性名
                property_value_unique[property_name[value_key]] = [];
                property_name_val_count[property_name[value_key]] = 0;
            }
            property_value_unique[property_name[value_key]].push(property_value[value_key]);//去重的属性名对应的属性值
            property_name_val_count[property_name[value_key]]++;//去重后的属性名有几个值
        };

        //总组合数 为各个属性 拥有值个数的 累乘
        for (var num_p in property_name_val_count) {
            var_total_group = property_name_val_count[num_p] * var_total_group;
        }


        var ceng = 0;           //当前第几层
        var pipei = [];         //匹配的结果值
        var ceng_old = 'X';     //上次循环时的层数
        var this_ceng_l_j = 1;  //当前层属性值个数 与 过往层个数累乘  var_total_group➗this_ceng_l_j=本层每种结果可出现重复的次数

        //根据不同属性个数 property_name_unique.length 分为 property_name_unique.length 层

        for (var time = 0; time < var_total_group * property_name_unique.length; time++) {
            ceng = Math.floor(time / var_total_group);
            if (ceng == 0) {
                //第一层初始化字符串
                pipei[time % var_total_group] = '';
            }
            //这一层拥有属性值得个数
            this_ceng_l = property_value_unique[property_name_unique[ceng]].length;

            //当前层 属性值个数 与 过往层个数 累乘
            if (ceng_old != ceng) {
                this_ceng_l_j = this_ceng_l * this_ceng_l_j;
            }

            for (var ceng_t_n = 0; ceng_t_n < this_ceng_l; ceng_t_n++) {
                temp_str = pipei[time % var_total_group] + property_name_unique[ceng] + '--' + property_value_unique[property_name_unique[ceng]][ceng_t_n] + '||';
                //这种结果可出现重复的次数是否达到最大值
                if (in_array_count(temp_str, pipei) < var_total_group / this_ceng_l_j) {
                    pipei[time % var_total_group] = temp_str;
                    break;
                }
            }
            ceng_old = ceng;
        }

        return pipei;

    }

</script>

<?php

/**
 * 爬取 国外高校犯罪信息数据
 * Class SchoolSpider
 */
class SchoolSpider
{
    private $schoolName;
    //学校名检查api
    private $schoolApi = 'https://ope.ed.gov/campussafety/api/institution/names?filter=';
    //校区
    private $schoolSerachApi = 'https://ope.ed.gov/campussafety/api/institution/search';
    //数据  校区
    private $dataApi = 'https://ope.ed.gov/campussafety/api/campus/';
    //整个学校
    private $totalApi = 'https://ope.ed.gov/campussafety/api/institution/';
    //校区
    private $campisesData;
    //详情数据
    private $dataDetail;
    //详情数据
    public $error;

    public $result=[];

    public function __construct()
    {
        //请求时间不限
        set_time_limit(0);
    }

    public function spiderData($schoolName)
    {
        $this->schoolName = $schoolName;
        $this->result['school_name'] = $this->schoolName;
        if (!$this->checkName() || !$this->getCampises()) {
            return ['code' => -1, 'msg' => $this->error];
        }
        //总的数据
        $this->formatData($this->campisesData);
        //各个分校区数据
        $this->formatData($this->campisesData[0]['Campuses'],1);

        return ['code' => 0, 'msg' => $this->error,'data'=>$this->result];
    }


    /**
     * @name  数据解析
     * @param $campisesData 数据
     * @param $isCampise  是否是分校区数据
     */
    private function formatData($campisesData,$isCampise=false){

        foreach ($campisesData as $campise) {
            if($isCampise){
                //校区名
                $res = $this->getDetailCampuse($campise['UnitID']);
            }else{
                $res = $this->getDetail($campise['UnitID']);
            }

            if(!$res||isset($this->result[$campise['UnitID']])){
                continue;
            }
            $dataDetail = $this->dataDetail;
            //校名
            $this->result[$campise['UnitID']]['campuse'] = $campise['Name'];
            foreach ($dataDetail as $k => $data) {
                if ($k == 0)
                    continue;
                $this->result[$campise['UnitID']]['data'][] = empty($data) ? [] : array_column($data['Cells'], 'Html');
            }
        }
    }

    /**
     * 检查学校名
     * @return array
     */
    private function checkName()
    {
        $school = json_decode($this->httpRequest($this->schoolApi . urlencode($this->schoolName)), true);
        if (!isset($school[0]['Code']) || !$school[0]['Code']) {
            $this->error = '检查学校名 error';
            return false;
        }
        return true;
    }

    /**
     * 获取校区
     * @return array
     */
    private function getCampises()
    {
        $schoolSerachData = "{\"name\":\"" . $this->schoolName . "\",\"city\":\"\",\"state\":[],\"country\":[],\"countryNames\":[],\"institutionType\":[],\"institutionProgram\":[],\"campusLocation\":\"-1\",\"onlyResidentialCampuses\":false,\"enrollmentRange\":[],\"sort\":\"name\",\"sortDirection\":\"asc\",\"all\":false,\"pageNumber\":0,\"fromFavorites\":false}";
        //获取校区列表
        $campisesData = json_decode($this->httpRequest($this->schoolSerachApi, $schoolSerachData), true);

        if (!isset($campisesData['Results'][0]['Campuses']) || empty($campisesData['Results'][0]['Campuses'])) {
            $this->error = '获取校区 error';
            return false;
        }
        $this->campisesData = $campisesData['Results'];
        return true;
    }

    /**
     * @name 获取全部详情
     * @param $unitId
     * @return array
     */
    private function getDetail($unitId)
    {

        $dataDetail = json_decode($this->httpRequest($this->totalApi . $unitId), true);

        if (!isset($dataDetail['Groups'][0]['Screens'][0]['Rows']) || empty($dataDetail['Groups'][0]['Screens'][0]['Rows'])) {
            $this->error = '获取详情 error';
            return false;
        }
        $this->dataDetail = $dataDetail['Groups'][0]['Screens'][0]['Rows'];
        return true;
    }

    /**
     * @name 获取分校区详情
     * @param $unitId
     * @return array
     */
    private function getDetailCampuse($unitId)
    {

        $dataDetail = json_decode($this->httpRequest($this->dataApi . $unitId), true);

        if (!isset($dataDetail['Groups'][0]['Screens'][0]['Rows']) || empty($dataDetail['Groups'][0]['Screens'][0]['Rows'])) {
            $this->error = '获取详情 error';
            return false;
        }
        $this->dataDetail = $dataDetail['Groups'][0]['Screens'][0]['Rows'];
        return true;
    }

    /**
     * @name 请求方法
     * @param $url
     * @param $data
     * @return false|mixed|string
     */
    private function httpRequest($url, $data = '')
    {
        $curl = curl_init();
        $method = $data ? 'POST' : 'GET';
        curl_setopt_array($curl, array(
            CURLOPT_URL => $url,
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_ENCODING => "",
            CURLOPT_MAXREDIRS => 10,
            CURLOPT_TIMEOUT => 30,
            CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
            CURLOPT_CUSTOMREQUEST => $method,
            CURLOPT_POSTFIELDS => $data,
            CURLOPT_HTTPHEADER => array(
                "Content-Type: application/json;charset=UTF-8",
                "Postman-Token: a30f51b0-cc09-4bfb-801e-830a931994f4",
                "cache-control: no-cache"
            ),
        ));

        $response = curl_exec($curl);
        $err = curl_error($curl);

        curl_close($curl);

        if ($err) {
            echo "cURL Error #:" . $err;
            return json_encode([]);
        } else {
            return strip_tags($response);
        }
    }
}

$a = new  SchoolSpider();
$r = $a->spiderData('Harvard University');
echo  json_encode($r);